Giter VIP home page Giter VIP logo

borg_ynh's Introduction

Borg Backup for YunoHost

Integration level Working status Maintenance status

Install Borg Backup with YunoHost

Lire ce readme en français.

This package allows you to install Borg Backup quickly and simply on a YunoHost server. If you don't have YunoHost, please consult the guide to learn how to install it.

Overview

Backup your server on a host server using Borg

Shipped version: 1.1.16~ynh30

Disclaimers / important information

⚠️ NB. : This doc is partially obsolete and should be reworked! ⚠️

How to backup your server with this app?

You want to backup a critical "guest" Server A onto a remote "host" Server B, you need:

  • Domain name of server B: host.serverb
  • Name of the server B SSH user (to be created by borgserver) for connection from Server A: borgservera
  • Strong passphrase to encrypt your backups on host Server B. And to restore your backups!!
  • IDs of YunoHost apps you want to backup
  • Regular time schedule for your backups, see below
  • Install Borg Backup App (borg) on guest Server A
  • Install Borg Server App (borgserver) on host Server B
  • Save the passphrase in another place than your server. Without the passphrase, you won't be able to restore data.

You should received an email after the first backup succeeded.

Set up Borg Backup App on guest Server A

Firstly, set up the Borg Backup App (borg) on the guest Server A you want to backup:

$ yunohost app install borg
In which borg repository location do you want to backup your files ?: [email protected]:/remote/repository
Provide a strong passphrase to encrypt your backups. No blank space:
Should Borg backup your YunoHost configuration? [yes | no] (default: yes):
Should Borg backup emails and user home directory? [yes | no] (default: yes):
Which apps should Borg backup ? (default: all):
With which regular time schedule should the backups be performed? (see systemd OnCalendar format) (default: Daily):
Do you want admin to receive mail notifications on backups ? [always | errors_only | never]: never

Syntax to define a backup time schedule

You can schedule regular backups at specific time. Only one regular time schedule is possible for one borg instance, see below for workaround. Some examples:

Information generated by Borg Backup

At the end of the installation, the Borg Backup App (borg) displays the SSH public key and the SSH user to give to the person who has access to the host Server B and will set up Borg Server App.

You should now install the "Borg Server" app on host.serverb and fill questions like this:
User: servera
Public key: ssh-ed25519 AAAA[...] [email protected]

This information is also sent by email to the admin of guest Server A. If you don't find the email and you don't see the message in the log bar you can find the SSH public key with this command:

$ cat /root/.ssh/id_borg_ed25519.pub
ssh-ed25519 AAAA[...] [email protected]

Set up Borg Server App on host Server B

Secondly, set up the Borg Server App (borgserver) on the host Server B that will store your backups:

$ yunohost app install borgserver
Indicate the ssh user to create: servera
Indicate the public key given by Borg Backup app (borg) setup: ssh-ed25519 AAAA[...] [email protected]
Indicate the storage quota: 5G

Test the Borg Apps setup

At this step your backup should run at the scheduled time. Note that the first backup can take very long, as much data has to be copied through ssh. Following backups are incremental: only newly generated data since last backup will be copied.

If you want to test correct Borg Apps setup before scheduled time, you can start a backup manually on guest Server A:

$ systemctl start borg

Next you can check presence of your backup repository on host Server B:

$ BORG_RSH="ssh -i /root/.ssh/id_borg_ed25519 -oStrictHostKeyChecking=yes " borg list [email protected]:~/backup

You will need the passphrase to run borg commands on the backup repository created on the host Server B.

Check regularly your backup

If you want to be sure to be able to restore your server, you should try to restore regularly the archives. But this process is quite time consumming.

You should at least:

  • Keep your apps up to date (if apps are too old, they could be difficult to restore on a more recent recent version)
  • Check regularly the presence of info.json and db.sql or dump.sql in your apps archives
borg list ./::ARCHIVE_NAME | grep info.json
borg list ./::ARCHIVE_NAME | grep db.sql
borg list ./::ARCHIVE_NAME | grep dump.sql
  • Be sure to have your passphrase available even if your server is completely broken

How to restore a complete system

For infos on restoring process, check this yunohost forum thread and that one, also using Borg with sshkeys, the borg extract documentation, and this general tutorial on Borg Backup.

In the following explanations:

  • the server to backup/restore will be called: yuno
  • the remote server that receives and store the back will be called: rem
  • rem is accessible at the domain rem.tld
  • the remote user on rem which owns the Borg backups will be called yurem
  • backup files will be stored in rem in the directory: /home/yurem/backup

Overview

The idea here, if you need to restore a whole yunohost system is:

  1. Install a new Debian VM
  2. Install YunoHost in it the usual way
  3. Go through YunoHost postinstall (parameters you will supply are not crucial, as they will be replaced by the restore)
  4. Install Borg
  5. Setup rem to accept ssh connections from yuno
  6. Use Borg to import backups from rem to yuno
  7. Restore Borg backups with the yunohost backup restore command, first config, then data, then each app one at a time
  8. Remove the Borg app and restore it

Make it possible for yuno to connect to rem with Borg

At this stage, we will assume that yuno is a freshly installed YunoHost (based on Buster in my case). You should also have performed the YunoHost postinstall.

If you don't want to restore the whole system, just some apps, you can skip some of the steps below.

Install the Borg YunoHost app in yuno

The idea here is just to install Borg, not in order to create backups, but only to use Borg commands to import remote backups.

So for example, you can install it doing the following:

sudo yunohost app install borg -a "server=rem.tld&ssh_user=yurem&conf=0&data=0&apps=hextris&on_calendar=2:30"

Make sure that rem accepts ssh connections from yuno

In yuno you will need to get the ssh key that borg just created while installing: sudo cat /root/.ssh/id_borg_ed25519.pub, copy it to clipboard.

Connect via ssh to rem, go to /home/yurem/.ssh/authorized_keys, and past the Borg public key you got at previous step.

Now to make sure this worked, you can try to SSH from yuno to rem. In yuno : ssh -i /root/.ssh/id_borg_ed25519 [email protected] . If you can get into rem , without it prompting for a password, then you're good to continue :)

Restore backups to yuno

⚠️ For the commands in the following section to work, you will need to be root in yuno (you can become root running sudo su).

⚠️ Restoration of backups can take quite a while, you'd better do them in a separate process, so that it doesn't stop if your terminal session gets closed. For this, you can for example use tmux.

In yuno now, you should be able to list backups in rem with the following command:

[email protected]:/home/yurem/backup
BORG_RSH="ssh -i /root/.ssh/id_borg_ed25519 -oStrictHostKeyChecking=yes " borg list $SRV

You can then reimport one to yuno with:

BORG_RSH="ssh -i /root/.ssh/id_borg_ed25519 -oStrictHostKeyChecking=yes " borg export-tar $SRV::auto_BACKUP_NAME /home/yunohost.backup/archives/auto_BACKUP_NAME.tar.gz

And then restore the archive in yuno with:

yunohost backup restore auto_BACKUP_NAME --system # for config and data backups
yunohost backup restore auto_BACKUP_NAME --apps # for other backups (=apps)

And Nextcloud? It's super heavy!!

For Nextcloud, the best is probably to reimport the backup without the data. And to import the data manually.

For that, you can do the following (as root):

[email protected]:/home/yurem/backup

# export the app without data
BORG_RSH="ssh -i /root/.ssh/id_borg_ed25519 -oStrictHostKeyChecking=yes " borg export-tar -e apps/nextcloud/backup/home/yunohost.app $SRV::auto_nextcloud_XX_XX_XX_XX:XX /home/yunohost.backup/archives/auto_nextcloud_XX_XX_XX_XX:XX.tar.gz

# extract the data from the backup to the nextcloud folder
cd /home/yunohost.app/nextcloud
BORG_RSH="ssh -i /root/.ssh/id_borg_ed25519 -oStrictHostKeyChecking=yes " borg extract $SRV::auto_nextcloud_XX_XX_XX_XX:XX apps/nextcloud/backup/home/yunohost.app/nextcloud/
mv apps/nextcloud/backup/home/yunohost.app/nextcloud/data data
rm -r apps

# now you can simply restore nextcloud app
yunohost backup restore auto_nextcloud_XX_XX_XX_XX:XX --apps

Restore Borg

Once you've restored the whole system, you will probably want to restore the Borg app as well.

For that, remove the "dummy" Borg you installed to do the restoration, and restore Borg the same ways as for other apps:

sudo yunohost app remove borg
sudo yunohost backup restore auto_borg_XX_XX_XX_XX:XX --apps

Tips

Edit the list of YunoHost apps to backup

yunohost app setting borg apps -v "nextcloud,wordpress"

Other usefull borg commands

Get the storage space used by the backup repository on the host server borg info /home/servera/backup

Backup YunoHost apps with different criticallity levels

If you want to backup your guest server:

  • with different YunoHost apps
  • at different regular time schedule
  • on different host servers

Then you can set up multiple instances of the Borg Apps on same servers. For instance:

  • Borg Backup instance borg: backup nextcloud daily on host Server B
  • Borg Backup instance borg__2: backup all other YunoHost apps weekly on host Server C

Settings location

Settings of apps are in /etc/yunohost/apps/*/settings.yml, so are the settings for borg_ynh as well. Be careful though to not savagely manually edit them if you don't know what you're doing.

Documentation and resources

Developer info

Please send your pull request to the testing branch.

To try the testing branch, please proceed like that.

sudo yunohost app install https://github.com/YunoHost-Apps/borg_ynh/tree/testing --debug
or
sudo yunohost app upgrade borg -u https://github.com/YunoHost-Apps/borg_ynh/tree/testing --debug

More info regarding app packaging: https://yunohost.org/packaging_apps

borg_ynh's People

Contributors

alexaubin avatar autra avatar bleuchtang avatar clemorange22 avatar eauchat avatar ericgaspar avatar gredin67 avatar jaxom99 avatar jocelyndelalande avatar kvl159 avatar leosw42 avatar ndarilek avatar nicofrand avatar utzer avatar yunohost-bot avatar zamentur avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar avatar

Forkers

jocelyndelalande nicofrand gabrielcossette autra ashemsay gredin67 kvl159 heartshare utzer eauchat fflorent bleuchtang ndarilek clemorange22 jaxom99 shukon orhtej2 jmthackett supermamie limezy

borg_ynh's Issues

Problem backing up large user folders

Hi,
after some searching I believe that for some reason my user folders cannot be backed up by borg (anymore).
I think all app backups and so on work without a problem (based on listing the backups) but the last backup of my user directory is from 2021-06-25 (when the backups still were not prefixed with a "_").
I am not sure why but it could be that the problem is related to one of the user directories being rather large.

My borg emails begin with

Creating a backup archive from the collected files...
The archive will contain about 340.5KiB of data.
Backup created
name: auto_conf
results: 
  apps: 
  system: 
    conf_ldap: Success
    conf_manually_modified_files: Success
    conf_ynh_certs: Success
    conf_ynh_settings: Success
size: 348708
Creating a backup archive from the collected files...
The archive will contain about 16.8GiB of data.
The operation 'Create a backup archive' could not be completed. Please share the full log of this operation using the command 'yunohost log share 20210921-210443-backup_create' to get help
Collecting files to be backed up for borg...
Loading installation settings...
Declaring files to be backed up...
Backup script completed for borg. (YunoHost will then actually copy those files to the archive).
Creating a backup archive from the collected files...
The archive will contain about 111.1KiB of data.
Backup created
name: auto_borg
results: 
  apps: 
    borg: Success
  system: 
size: 113773

for the full log see here https://paste.yunohost.org/raw/gefipadini for the full log in particular

2021-09-22 01:46:05,053: DEBUG - ++ date +%Y-%m-%d_%H:%M:%S
2021-09-22 01:46:05,067: DEBUG - + sed -e 's/^/[2021-09-22_01:46:05] /'
2021-09-22 07:34:01,690: ERROR - Could not run script: /etc/yunohost/hooks.d/backup_method/05-borg_app
Traceback (most recent call last):
  File "/usr/lib/moulinette/yunohost/hook.py", line 307, in hook_callback
    path, args=hook_args, chdir=chdir, env=env, raise_on_error=True
  File "/usr/lib/moulinette/yunohost/hook.py", line 393, in hook_exec
    raise YunohostError("hook_exec_failed", path=path)
yunohost.utils.error.YunohostError: Could not run script: /etc/yunohost/hooks.d/backup_method/05-borg_app

Additionally at the end I get a lot of

Failed to format translated string 'backup_method_custom_finished': 'Custom backup method '{method}' finished' with arguments '()' and '{}, raising error: KeyError('method') (don't panic this is just a warning)
Failed to format translatable string 'backup_method_custom_finished': 'Custom backup method '{method}' finished' with arguments '()' and '{}', raising  error: KeyError('method') (don't panic this is just a warning)
Failed to format translated string 'backup_applying_method_custom': 'Calling the custom backup method '{method}'...' with arguments '()' and '{}, raising error: KeyError('method') (don't panic this is just a warning)
Failed to format translatable string 'backup_applying_method_custom': 'Calling the custom backup method '{method}'...' with arguments '()' and '{}', raising  error: KeyError('method') (don't panic this is just a warning)

warnings but I am not sure whether they are related.

Stop backing up on reboot

Describe the bug

Borg backup at each start, even reboot...

Steps to reproduce

  • reboot yunohost
  • borg backup on startup
  • takes all resources (cpu, bandwith...)

Expected behavior

Borg backup every week (in the middle of a night, no disturbance). It is enough for me. I don't need it to backup also everytime I reboot my server. I'd like to be able to stop the option to backup on startup.

do_known_hosts: hostkeys_foreach failed: No such file or directory

I tried to setup Borg following the instructions provided, however it keeps failing.
Setup: Yunohost 4.0.8 Buster, tested on 2 servers (same results) : 1 upgraded from Yunohost 3.X and 1 fresh installed 4.X

At first, the install ran into some issues, the package Cython (pip) was missing, I was able to solve this issue by installing it manually, however would it be possible to fix it in the ynh package ?

Here are the details regarding Borg not working using the following command:
$ sudo yunohost app install borgserver

Here is the log from journal :

-- The unit home-yunohost.backup-tmp-auto_rainloop-apps-rainloop-backup-var-www-rainloop.mount has successfully entered the 'dead' state.
DATE server.to.backup backup-with-borg[001]: Custom backup method could not get past the 'backup' step
DATE server.to.backup backup-with-borg[001]: Collecting files to be backed up for wordpress...
DATE server.to.backup backup-with-borg[001]: [++++................] > Loading installation settings...
DATE server.to.backup backup-with-borg[001]: [####++++............] > Backing up the main app directory...
DATE server.to.backup backup-with-borg[001]: [########++..........] > Backing up nginx web server configuration...
DATE server.to.backup backup-with-borg[001]: [##########++........] > Backing up php-fpm configuration...
DATE server.to.backup backup-with-borg[001]: [############++++....] > Backing up the MySQL database...
DATE server.to.backup backup-with-borg[001]: [################++..] > Backing up fail2ban configuration...
DATE server.to.backup backup-with-borg[001]: Creating a backup archive from the collected files...
DATE server.to.backup backup-with-borg[001]: do_known_hosts: hostkeys_foreach failed: No such file or directory
DATE server.to.backup backup-with-borg[001]: do_known_hosts: hostkeys_foreach failed: No such file or directory
DATE server.to.backup backup-with-borg[001]: Could not run script: /etc/yunohost/hooks.d/backup_method/05-borg_app
DATE server.to.backup systemd[002]: home-yunohost.backup-tmp-auto_wordpress-apps-wordpress-backup-var-www-wordpress.mount: Succeeded.
-- Subject: Unit succeeded
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- The unit UNIT has successfully entered the 'dead' state.
DATE server.to.backup systemd[1]: home-yunohost.backup-tmp-auto_wordpress-apps-wordpress-backup-var-www-wordpress.mount: Succeeded.
-- Subject: Unit succeeded
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- The unit home-yunohost.backup-tmp-auto_wordpress-apps-wordpress-backup-var-www-wordpress.mount has successfully entered the 'dead' state.
DATE server.to.backup backup-with-borg[001]: Custom backup method could not get past the 'backup' step
DATE server.to.backup systemd[1]: borg.service: Main process exited, code=exited, status=1/FAILURE
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- An ExecStart= process belonging to unit borg.service has exited.
--
-- The process' exit code is 'exited' and its exit status is 1.
DATE server.to.backup systemd[1]: borg.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- The unit borg.service has entered the 'failed' state with result 'exit-code'.
DATE server.to.backup systemd[1]: Failed to start Run backup borg.
-- Subject: A start job for unit borg.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- A start job for unit borg.service has finished with a failure.

The service status :

● borg.service - Run backup borg
   Loaded: loaded (/etc/systemd/system/borg.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since DATE UTC; 3min 7s ago
  Process: 001 ExecStart=/usr/local/bin/backup-with-borg (code=exited, status=1/FAILURE)
 Main PID: 001 (code=exited, status=1/FAILURE)

DATE server.to.backup backup-with-borg[001]: [############++++....] > Backing up the MySQL database...
DATE server.to.backup backup-with-borg[001]: [################++..] > Backing up fail2ban configuration...
DATE server.to.backup backup-with-borg[001]: Creating a backup archive from the collected files...
DATE server.to.backup backup-with-borg[001]: do_known_hosts: hostkeys_foreach failed: No such file or directory
DATE server.to.backup backup-with-borg[001]: do_known_hosts: hostkeys_foreach failed: No such file or directory
DATE server.to.backup backup-with-borg[001]: Could not run script: /etc/yunohost/hooks.d/backup_method/05-borg_app
DATE server.to.backup backup-with-borg[001]: Custom backup method could not get past the 'backup' step
DATE server.to.backup systemd[1]: borg.service: Main process exited, code=exited, status=1/FAILURE
DATE server.to.backup systemd[1]: borg.service: Failed with result 'exit-code'.
DATE server.to.backup systemd[1]: Failed to start Run backup borg.

And the error log from borg :

./05-borg_app: line 31: borg: command not found
./05-borg_app: line 44: borg: command not found

It seems that the borg command isn't working as well, any idea where could be the issue ?

Allow to chose the repository path.

Hi,

thanks a lot for this script, I do hope I'll be able to make it work as expected and finally be able to backup my data.

I am using borgbase.com as a provider (instead of using a "server B" width borgserver_ynh installed). They provide a repo path like [email protected]:repo.

To be able to instantiate the creation I had to edit this line: and replace the ~/backup/ by ~/repo/.

It would be great if the user could select its own path (like for the server name etc.) during the installation.

Yunohost diagnosis reports borg service status to be dead

Maybe I configured something wrong, but in my yunohost server, ynh diagnosis is raising errors saying that borg service is dead.

From what I understand, it makes sense that the service is dead, since it's starting only when it's making a backup.
It would be really nice indeed to know when backups are failing, but maybe the yunohost diagnosis should behave differently for borg_ynh service than other services. Maybe it should consider "dead" status to be ok, and raise an error if the status of the service is "failed".

But maybe I misunderstood something. Or I can imagine that it's not trivial to implement.

feature request: send one backup on multiple locations

It could be cool to do the backup once (I mean, all the work borg does before sending the backup to the destination) and send it to two (or more) different locations. For example, you can do a backup on a local hard disk and on a distant location.

Add an option to backup the yunohost.multimedia repository

I'm using the yunohost.multimedia repository as the main place where I store my personal data (it's easier to mount yunohost.multimedia as an encrypted partition than the whole /home folder), and borg to make my backups. It would be nice to be able to say to borg to save the yunohost.multimedia folder then.

Folder already exists: '/home/yunohost.backup/tmp/***'

I tried to manually start a backup via sudo systemctl start borg.service

I got the log in real time with sudo journalctl -fu borg.service

It failed with :

   filesystem.mkdir(self.work_dir, 0o750, parents=True, uid='admin')
File "/usr/lib/python2.7/dist-packages/moulinette/utils/filesystem.py", line 273, in mkdir
   raise OSError(errno.EEXIST, m18n.g("folder_exists", path=path))
OSError: [Errno 17] Folder already exists: '/home/yunohost.backup/tmp/auto_data'
Collecting files to be backed up for borg...
Creating a backup archive from the collected files...
Could not run script: /etc/yunohost/hooks.d/backup_method/05-borg_app
Custom backup method could not get past the 'backup' step
Collecting files to be backed up for borgserver...
Creating a backup archive from the collected files...
Could not run script: /etc/yunohost/hooks.d/backup_method/05-borg_app
Custom backup method could not get past the 'backup' step
Traceback (most recent call last):
  File "/usr/bin/yunohost", line 218, in <module>
  timeout=opts.timeout,
  File "/usr/lib/python2.7/dist-packages/moulinette/__init__.py", line 149, in cli
   moulinette.run(args, output_as=output_as, password=password, timeout=timeout)
 File "/usr/lib/python2.7/dist-packages/moulinette/interfaces/cli.py", line 469, in run
   ret = self.actionsmap.process(args, timeout=timeout)
  File "/usr/lib/python2.7/dist-packages/moulinette/actionsmap.py", line 588, in process
    return func(**arguments)
   File "/usr/lib/moulinette/yunohost/backup.py", line 2172, in backup_create
   backup_manager = BackupManager(name, description)
  File "/usr/lib/moulinette/yunohost/backup.py", line 277, in __init__
   self._init_work_dir()
 File "/usr/lib/moulinette/yunohost/backup.py", line 348, in _init_work_dir
    filesystem.mkdir(self.work_dir, 0o750, parents=True, uid='admin')
  File "/usr/lib/python2.7/dist-packages/moulinette/utils/filesystem.py", line 273, in mkdir
    raise OSError(errno.EEXIST, m18n.g("folder_exists", path=path))
 OSError: [Errno 17] Folder already exists: '/home/yunohost.backup/tmp/auto_dokuwiki'

followed by the same for all installed applications.
I tried to do a normal Yunohost backup and it works, so I don't understand why it fails with Borg app.

Thanks!

Possibility to exclude/include nextcloud data from backup

As I evoked in this topic on the forum, I recently discovered that my server hadn't been backed up for quite a while, because of a lack of space in the target server.

I'm trying to achieve two things to avoid this problem:

  • avoid backing up nextcloud data
  • getting some warnings on backup failures

I believe the first point can be achieved by fiddling around with the list of directories that borg is backing up, but I don't want to break the way borg_ynh makes it easy to add/remove some apps to the backup. Anyone has some advice on a good way to do that?
Also would be very grateful for suggestions on how to achieve the second point.

Thanks a lot for any advice and for this great app.

Feature Request: Backup Nextcloud data

I just did a reinstall of the borg app for yunohost and (luckily) realized that the Nextcloud data isn't actually getting backed up! I thought the main point of borg backup for yunohost was being able to do large, frequent data backups?

I think that another case will need to be added to backup-with-borg.j2 that for Nextcloud, it does it twice, once with backup_core_only set or not set. Probably need to add that to configuration questions during install.

Custom backup method failure on 'backup' step

I am trying to configure borg to backup to my repo at borgbase.com.

root@domain:~# yunohost app install https://github.com/YunoHost-Apps/borg_ynh
Indicate the server where you want put your backups: edited.repo.borgbase.com
Indicate the ssh user to use to connect on this server: edited
You are now about to define a new user password. The password should be at least 8 characters - though it is good practice to use longer password (i.e. a passphrase) and/or to use various kind of characters (uppercase, lowercase, digits and special characters).
Indicate a strong passphrase, that you will keep preciously if you want to be able to use your backups:
Would you like to backup your YunoHost configuration ? [yes | no] (default: yes): yes
Would you like to backup mails and user home directory ? [yes | no] (default: yes): yes
Which apps would you backup (list separated by comma or 'all') ? (default: all): all
Indicate the backup frequency (see systemd OnCalendar format) (default: Daily): Daily
Warning: Created symlink /etc/systemd/system/multi-user.target.wants/borg.service → /etc/systemd/system/borg.service.
Warning: Created symlink /etc/systemd/system/timers.target.wants/borg.timer → /etc/systemd/system/borg.timer.
Success! The SSOwat configuration has been generated
Success! Installation complete

Setup went fine, but when I go to start borg to ensure a successful backup, I get an error:

Jan 15 14:21:32 domain.tld systemd[1]: Starting Run backup borg...
Jan 15 14:21:35 domain.tld backup-with-borg[1820]: Script execution failed: /etc/yunohost/hooks.d/backup_method/05-borg_app
Jan 15 14:21:35 domain.tld backup-with-borg[1820]: Custom backup method failure on 'backup' step
Jan 15 14:21:36 domain.tld slapd[816]: <= mdb_equality_candidates: (cn) not indexed
Jan 15 14:21:36 domain.tld slapd[816]: <= mdb_equality_candidates: (sudoUser) not indexed
Jan 15 14:21:36 domain.tld slapd[816]: <= mdb_equality_candidates: (sudoUser) not indexed
Jan 15 14:21:36 domain.tld slapd[816]: <= mdb_equality_candidates: (sudoUser) not indexed
Jan 15 14:21:36 domain.tld slapd[816]: <= mdb_equality_candidates: (sudoUser) not indexed
Jan 15 14:21:36 domain.tld slapd[816]: <= mdb_substring_candidates: (sudoUser) not indexed
Jan 15 14:21:36 domain.tld sudo[1940]:     root : TTY=unknown ; PWD=/home/yunohost.backup/tmp/auto_nextcloud/apps/nextcloud/backup ; USER=root ; COMMAND=/usr/bin/yunohost app setting nextcloud final_path --output-as plain --quiet
Jan 15 14:21:36 domain.tld sudo[1940]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jan 15 14:21:36 domain.tld sudo[1940]: pam_unix(sudo:session): session closed for user root
Jan 15 14:21:36 domain.tld slapd[816]: <= mdb_equality_candidates: (cn) not indexed
Jan 15 14:21:36 domain.tld slapd[816]: <= mdb_equality_candidates: (sudoUser) not indexed
Jan 15 14:21:36 domain.tld slapd[816]: <= mdb_equality_candidates: (sudoUser) not indexed
Jan 15 14:21:36 domain.tld slapd[816]: <= mdb_equality_candidates: (sudoUser) not indexed
Jan 15 14:21:36 domain.tld slapd[816]: <= mdb_equality_candidates: (sudoUser) not indexed
Jan 15 14:21:36 domain.tld slapd[816]: <= mdb_substring_candidates: (sudoUser) not indexed
Jan 15 14:21:36 domain.tld sudo[1943]:     root : TTY=unknown ; PWD=/home/yunohost.backup/tmp/auto_nextcloud/apps/nextcloud/backup ; USER=root ; COMMAND=/usr/bin/yunohost app setting nextcloud domain --output-as plain --quiet
Jan 15 14:21:36 domain.tld sudo[1943]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jan 15 14:21:37 domain.tld sudo[1943]: pam_unix(sudo:session): session closed for user root
Jan 15 14:21:37 domain.tld slapd[816]: <= mdb_equality_candidates: (cn) not indexed
Jan 15 14:21:37 domain.tld slapd[816]: <= mdb_equality_candidates: (sudoUser) not indexed
Jan 15 14:21:37 domain.tld slapd[816]: <= mdb_equality_candidates: (sudoUser) not indexed
Jan 15 14:21:37 domain.tld slapd[816]: <= mdb_equality_candidates: (sudoUser) not indexed
Jan 15 14:21:37 domain.tld slapd[816]: <= mdb_equality_candidates: (sudoUser) not indexed
Jan 15 14:21:37 domain.tld slapd[816]: <= mdb_substring_candidates: (sudoUser) not indexed
Jan 15 14:21:37 domain.tld sudo[1946]:     root : TTY=unknown ; PWD=/home/yunohost.backup/tmp/auto_nextcloud/apps/nextcloud/backup ; USER=root ; COMMAND=/usr/bin/yunohost app setting nextcloud db_name --output-as plain --quiet
Jan 15 14:21:37 domain.tld sudo[1946]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jan 15 14:21:37 domain.tld sudo[1946]: pam_unix(sudo:session): session closed for user root
Jan 15 14:21:37 domain.tld slapd[816]: <= mdb_equality_candidates: (cn) not indexed
Jan 15 14:21:37 domain.tld slapd[816]: <= mdb_equality_candidates: (sudoUser) not indexed
Jan 15 14:21:37 domain.tld slapd[816]: <= mdb_equality_candidates: (sudoUser) not indexed
Jan 15 14:21:37 domain.tld slapd[816]: <= mdb_equality_candidates: (sudoUser) not indexed
Jan 15 14:21:37 domain.tld slapd[816]: <= mdb_equality_candidates: (sudoUser) not indexed
Jan 15 14:21:37 domain.tld slapd[816]: <= mdb_substring_candidates: (sudoUser) not indexed
Jan 15 14:21:37 domain.tld sudo[1984]:     root : TTY=unknown ; PWD=/home/yunohost.backup/tmp/auto_nextcloud/apps/nextcloud/backup ; USER=root ; COMMAND=/bin/cat /etc/yunohost/mysql
Jan 15 14:21:37 domain.tld sudo[1984]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jan 15 14:21:37 domain.tld sudo[1984]: pam_unix(sudo:session): session closed for user root
Jan 15 14:21:38 domain.tld slapd[816]: <= mdb_equality_candidates: (cn) not indexed
Jan 15 14:21:38 domain.tld slapd[816]: <= mdb_equality_candidates: (sudoUser) not indexed
Jan 15 14:21:38 domain.tld slapd[816]: <= mdb_equality_candidates: (sudoUser) not indexed
Jan 15 14:21:38 domain.tld slapd[816]: <= mdb_equality_candidates: (sudoUser) not indexed
Jan 15 14:21:38 domain.tld slapd[816]: <= mdb_equality_candidates: (sudoUser) not indexed
Jan 15 14:21:38 domain.tld slapd[816]: <= mdb_substring_candidates: (sudoUser) not indexed
Jan 15 14:21:38 domain.tld sudo[2008]:     root : TTY=unknown ; PWD=/home/yunohost.backup/tmp/auto_nextcloud/apps/nextcloud/backup ; USER=root ; COMMAND=/usr/bin/yunohost app setting nextcloud backup_core_only --output-as plain --quiet
Jan 15 14:21:38 domain.tld sudo[2008]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jan 15 14:21:38 domain.tld sudo[2008]: pam_unix(sudo:session): session closed for user root
Jan 15 14:21:43 domain.tld backup-with-borg[1820]: Script execution failed: /etc/yunohost/hooks.d/backup_method/05-borg_app
Jan 15 14:21:43 domain.tld backup-with-borg[1820]: Custom backup method failure on 'backup' step
Jan 15 14:21:44 domain.tld slapd[816]: <= mdb_equality_candidates: (cn) not indexed
Jan 15 14:21:44 domain.tld slapd[816]: <= mdb_equality_candidates: (sudoUser) not indexed
Jan 15 14:21:44 domain.tld slapd[816]: <= mdb_equality_candidates: (sudoUser) not indexed
Jan 15 14:21:44 domain.tld slapd[816]: <= mdb_equality_candidates: (sudoUser) not indexed
Jan 15 14:21:44 domain.tld slapd[816]: <= mdb_equality_candidates: (sudoUser) not indexed
Jan 15 14:21:44 domain.tld slapd[816]: <= mdb_substring_candidates: (sudoUser) not indexed
Jan 15 14:21:44 domain.tld sudo[2066]:     root : TTY=unknown ; PWD=/home/yunohost.backup/tmp/auto_rainloop/apps/rainloop/backup ; USER=root ; COMMAND=/usr/bin/yunohost app setting rainloop domain --output-as plain --quiet
Jan 15 14:21:44 domain.tld sudo[2066]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jan 15 14:21:44 domain.tld sudo[2066]: pam_unix(sudo:session): session closed for user root
Jan 15 14:21:44 domain.tld slapd[816]: <= mdb_equality_candidates: (cn) not indexed
Jan 15 14:21:44 domain.tld slapd[816]: <= mdb_equality_candidates: (sudoUser) not indexed
Jan 15 14:21:44 domain.tld slapd[816]: <= mdb_equality_candidates: (sudoUser) not indexed
Jan 15 14:21:44 domain.tld slapd[816]: <= mdb_equality_candidates: (sudoUser) not indexed
Jan 15 14:21:44 domain.tld slapd[816]: <= mdb_equality_candidates: (sudoUser) not indexed
Jan 15 14:21:44 domain.tld slapd[816]: <= mdb_substring_candidates: (sudoUser) not indexed
Jan 15 14:21:44 domain.tld sudo[2069]:     root : TTY=unknown ; PWD=/home/yunohost.backup/tmp/auto_rainloop/apps/rainloop/backup ; USER=root ; COMMAND=/usr/bin/yunohost app setting rainloop path --output-as plain --quiet
Jan 15 14:21:44 domain.tld sudo[2069]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jan 15 14:21:44 domain.tld sudo[2069]: pam_unix(sudo:session): session closed for user root
Jan 15 14:21:44 domain.tld slapd[816]: <= mdb_equality_candidates: (cn) not indexed
Jan 15 14:21:44 domain.tld slapd[816]: <= mdb_equality_candidates: (sudoUser) not indexed
Jan 15 14:21:44 domain.tld slapd[816]: <= mdb_equality_candidates: (sudoUser) not indexed
Jan 15 14:21:44 domain.tld slapd[816]: <= mdb_equality_candidates: (sudoUser) not indexed
Jan 15 14:21:44 domain.tld slapd[816]: <= mdb_equality_candidates: (sudoUser) not indexed
Jan 15 14:21:44 domain.tld slapd[816]: <= mdb_substring_candidates: (sudoUser) not indexed
Jan 15 14:21:44 domain.tld sudo[2072]:     root : TTY=unknown ; PWD=/home/yunohost.backup/tmp/auto_rainloop/apps/rainloop/backup ; USER=root ; COMMAND=/usr/bin/yunohost app setting rainloop mysqlpwd --output-as plain --quiet
Jan 15 14:21:44 domain.tld sudo[2072]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jan 15 14:21:44 domain.tld sudo[2072]: pam_unix(sudo:session): session closed for user root
Jan 15 14:21:46 domain.tld backup-with-borg[1820]: Script execution failed: /etc/yunohost/hooks.d/backup_method/05-borg_app
Jan 15 14:21:46 domain.tld backup-with-borg[1820]: Custom backup method failure on 'backup' step
Jan 15 14:21:46 domain.tld systemd[1]: borg.service: Main process exited, code=exited, status=5/NOTINSTALLED
Jan 15 14:21:46 domain.tld systemd[1]: borg.service: Failed with result 'exit-code'.
Jan 15 14:21:46 domain.tld systemd[1]: Failed to start Run backup borg.

Is it failing on the "yunohost app setting nextcloud backup_core_only" step?

How can I proceed with fixing this?

Thanks

---
Automatic diagnosis data from YunoHost
host: Debian 9.6
kernel: 4.18.0-0.bpo.1-amd64
packages: 
  yunohost: 
    repo: stable
    version: 3.3.3
  yunohost-admin: 
    repo: stable
    version: 3.3.1
  moulinette: 
    repo: stable
    version: 3.3.1
  ssowat: 
    repo: stable
    version: 3.3.2
backports: 
system: 
  disks: 
    vda1: Mounted on /, 146.2GiB (126.7GiB free)
  memory: 
    ram: 484.9MiB (168.2MiB free)
    swap: 509.0MiB (190.3MiB free)
nginx: 
  - nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
  - nginx: configuration file /etc/nginx/nginx.conf test is successful

Error not logged when `borg create` fails

Describe the bug

Sometimes, the borg create command fails in the backup_method script. I don't get any error log in /var/log/borg/, and I miss clues why the error occurs.

Context

NB: I use a distant borg repository which is not a Yunohost server (I tweak the value of the repository variable).

  • Hardware: VPS bought online / Old laptop or computer / Raspberry Pi at home / Internet Cube with VPN / Other ARM board / ...
  • YunoHost version: 4.2.8
  • I have access to my server: Through SSH + through the webadmin
  • Are you in a special context or did you perform some particular tweaking on your YunoHost instance?: no
    • If yes, please explain:
  • Using, or trying to install package version/branch:
  • If upgrading, current package version: can be found in the admin, or with yunohost app info $app_id

Steps to reproduce

Run systemctl start borg and ensure borg create fails I guess ¯_ (ツ)_/¯

Expected behavior

The error should be logged in /var/log/borg/...

Analysis

In lines invoking borg create and borg prune (here, here and here), I wonder why we redirect the output to /dev/null.

Cannot 2>&1 | log_with_timestamp be enough?

borg backup stops applications when doing backup & failing to backup application

First off, so happy to be testing and using borg via yunohost. It's a great piece of software and has saved my data from hardware failure a couple of times now. I really hope we can get some nice easy integration in yunohost.

On a yunohost with a single app, jellyfin, it seems that when the backup is run it stops the jellyfin application, fails to back it up and doesn't restart it.

root@host:~# BORG_RSH="ssh -i /root/.ssh/id_borg_ed25519 -oStrictHostKeyChecking=yes " borg extract user@server:repo::_auto_borg-2021-05-28_00:01 --list
Enter passphrase for key ssh://user@server/./repo: 
.
apps
apps/borg
apps/borg/backup
apps/borg/backup/usr
apps/borg/backup/usr/local
apps/borg/backup/usr/local/bin
apps/borg/backup/usr/local/bin/backup-with-borg
apps/borg/backup/etc
apps/borg/backup/etc/sudoers.d
apps/borg/backup/etc/sudoers.d/borg
apps/borg/backup/etc/systemd
apps/borg/backup/etc/systemd/system
apps/borg/backup/etc/systemd/system/borg.timer
apps/borg/backup/etc/systemd/system/borg.service
apps/borg/backup/etc/yunohost
apps/borg/backup/etc/yunohost/hooks.d
apps/borg/backup/etc/yunohost/hooks.d/backup_method
apps/borg/backup/etc/yunohost/hooks.d/backup_method/05-borg_app
apps/borg/backup/root
apps/borg/backup/root/.ssh
apps/borg/backup/root/.ssh/id_borg_ed25519
apps/borg/backup/root/.ssh/id_borg_ed25519.pub
apps/borg/settings
apps/borg/settings/settings.yml
apps/borg/settings/manifest.json
apps/borg/settings/permissions.yml
apps/borg/settings/scripts
apps/borg/settings/scripts/_common.sh
apps/borg/settings/scripts/install
apps/borg/settings/scripts/backup
apps/borg/settings/scripts/upgrade
apps/borg/settings/scripts/remove
apps/borg/settings/scripts/restore
apps/borg/settings/conf
apps/borg/settings/conf/systemd.timer
apps/borg/settings/conf/sudoer
apps/borg/settings/conf/backup-with-borg
apps/borg/settings/conf/backup_method
apps/borg/settings/conf/systemd.service
backup.csv
info.json

No jellyfin files seem to be there...

And the log of the backup itself goes :

root@etienne:~# yunohost log show 20210527-230143-backup_create
description: Create a backup archive
log_path: /var/log/yunohost/categories/operation/20210527-230143-backup_create.log
logs: 
  - 2021-05-28 00:01:49,311: DEBUG - + args_array=([s]=src_path= [d]=dest_path= [b]=is_big [m]=not_mandatory)
  - 2021-05-28 00:01:49,312: DEBUG - + local -A args_array
  - 2021-05-28 00:01:49,312: DEBUG - + local src_path
  - 2021-05-28 00:01:49,313: DEBUG - + local dest_path
  - 2021-05-28 00:01:49,314: DEBUG - + local is_big
  - 2021-05-28 00:01:49,314: DEBUG - + local not_mandatory
  - 2021-05-28 00:01:49,315: DEBUG - + ynh_handle_getopts_args --src_path=/etc/systemd/system/jellyfin.service
  - 2021-05-28 00:01:49,316: DEBUG - + set +o xtrace
  - 2021-05-28 00:01:49,417: DEBUG - + dest_path=
  - 2021-05-28 00:01:49,418: DEBUG - + is_big=0
  - 2021-05-28 00:01:49,419: DEBUG - + not_mandatory=0
  - 2021-05-28 00:01:49,421: DEBUG - + BACKUP_CORE_ONLY=0
  - 2021-05-28 00:01:49,421: DEBUG - + test -n jellyfin
  - 2021-05-28 00:01:49,422: DEBUG - ++ ynh_app_setting_get --app=jellyfin --key=do_not_backup_data
  - 2021-05-28 00:01:49,423: DEBUG - ++ local legacy_args=ak
  - 2021-05-28 00:01:49,424: DEBUG - ++ args_array=([a]=app= [k]=key=)
  - 2021-05-28 00:01:49,424: DEBUG - ++ local -A args_array
  - 2021-05-28 00:01:49,425: DEBUG - ++ local app
  - 2021-05-28 00:01:49,426: DEBUG - ++ local key
  - 2021-05-28 00:01:49,426: DEBUG - ++ ynh_handle_getopts_args --app=jellyfin --key=do_not_backup_data
  - 2021-05-28 00:01:49,427: DEBUG - ++ set +o xtrace
  - 2021-05-28 00:01:49,543: DEBUG - ++ [[ do_not_backup_data =~ (unprotected|protected|skipped)_ ]]
  - 2021-05-28 00:01:49,544: DEBUG - ++ ynh_app_setting get jellyfin do_not_backup_data
  - 2021-05-28 00:01:49,545: DEBUG - ++ set +o xtrace
  - 2021-05-28 00:01:49,833: DEBUG - + do_not_backup_data=
  - 2021-05-28 00:01:49,834: DEBUG - + '[' 0 -eq 1 ']'
  - 2021-05-28 00:01:49,835: DEBUG - + '[' '!' -e /etc/systemd/system/jellyfin.service ']'
  - 2021-05-28 00:01:49,836: DEBUG - + ynh_print_warn '--message=Source path '\''/etc/systemd/system/jellyfin.service'\'' does not exist'
  - 2021-05-28 00:01:49,836: DEBUG - + local legacy_args=m
  - 2021-05-28 00:01:49,837: DEBUG - + args_array=([m]=message=)
  - 2021-05-28 00:01:49,838: DEBUG - + local -A args_array
  - 2021-05-28 00:01:49,838: DEBUG - + local message
  - 2021-05-28 00:01:49,839: DEBUG - + ynh_handle_getopts_args '--message=Source path '\''/etc/systemd/system/jellyfin.service'\'' does not exist'
  - 2021-05-28 00:01:49,840: DEBUG - + set +o xtrace
  - 2021-05-28 00:01:49,859: DEBUG - + ynh_print_log 'Source path '\''/etc/systemd/system/jellyfin.service'\'' does not exist'
  - 2021-05-28 00:01:49,860: DEBUG - + echo -e 'Source path '\''/etc/systemd/system/jellyfin.service'\'' does not exist'
  - 2021-05-28 00:01:49,862: DEBUG - + '[' 0 == 0 ']'
  - 2021-05-28 00:01:49,868: WARNING - Source path '/etc/systemd/system/jellyfin.service' does not exist
  - 2021-05-28 00:01:49,870: DEBUG - + echo /etc/systemd/system/jellyfin.service
  - 2021-05-28 00:01:49,870: DEBUG - + grep --quiet /etc/fail2ban
  - 2021-05-28 00:01:49,871: DEBUG - + return 1
  - 2021-05-28 00:01:49,872: DEBUG - + ynh_exit_properly
  - 2021-05-28 00:01:49,872: DEBUG - + local exit_code=1
  - 2021-05-28 00:01:49,873: DEBUG - + rm -rf /var/cache/yunohost/download/
  - 2021-05-28 00:01:49,874: DEBUG - + '[' 1 -eq 0 ']'
  - 2021-05-28 00:01:49,875: DEBUG - + trap '' EXIT
  - 2021-05-28 00:01:49,876: DEBUG - + set +o errexit
  - 2021-05-28 00:01:49,876: DEBUG - + set +o nounset
  - 2021-05-28 00:01:49,877: DEBUG - + sleep 0.5
  - 2021-05-28 00:01:50,384: ERROR - Could not back up jellyfin
[snip]

Full https://paste.yunohost.org/raw/ritirijuxo

On thing I notice is that it's looking for the service file in the wrong place, it's actually in /lib/systemd/system/jellyfin.service

How to manually start a backup

Hi,

I want to test manually the backup. From the doc, I tried:

service borg start

But it doesn't work:

Job for borg.service failed because the control process exited with error code.
See "systemctl status borg.service" and "journalctl -xe" for details.

Then I tried: sudo /usr/local/bin/backup-with-borg

And it gives the error:

ls: cannot access '/etc/yunohost/hooks.d/backup/': No such file or directory
Warning: ls: cannot access '/etc/cron.d/yunohost*': No such file or directory
Error: Script execution failed: /etc/yunohost/hooks.d/backup_method/05-borg__2_app
Error: Custom backup method failure on 'backup' step
ls: cannot access '/etc/yunohost/hooks.d/backup/': No such file or directory
Error: Script execution failed: /etc/yunohost/hooks.d/backup_method/05-borg__2_app
Error: Custom backup method failure on 'backup' step
Error: Script execution failed: /etc/yunohost/hooks.d/backup_method/05-borg__2_app
Error: Custom backup method failure on 'backup' step
Error: Script execution failed: /etc/yunohost/hooks.d/backup_method/05-borg__2_app
Error: Custom backup method failure on 'backup' step

Would you have any suggestion? Many thanks!

Passphrase problem...

Hello,

When I do this command
BORG_RSH="ssh -i /root/.ssh/id_borg_ed25519 -oStrictHostKeyChecking=yes " borg list [email protected]:~/backup

as expected, it asks me the passphrase but when I write it, it doesn't work. I first thought it I made a mistake so I went to check in this file
sudo nano /etc/yunohost/hooks.d/backup_method/05-borg_app
and BORG_PASSPHRASE="xxxxx"
xxxxx is exactly what I wrote when the passphrase was requested.

Where am I wrong?
What should I do?

No output in /var/log/backup_borg.log

I do get errors in /var/log/backup_borg.err, but not in backup_borg.log. I am expecting to be able to tail that file and see what borg is doing. Right?

Also, it would be good to use the current_date variable to add timestamps to the log files.

Provide GUI for user control

After installing Borg Yunohost does not give the user the ability to view, change or update settings entered at install time. (For example - encryption passphrase, or remote servers.)

Ideally it should be possible to configure and update configurations for common backup scenarios (i.e. to remote server like Borgbase, or another yunhosted borg server) without needing to access the command line.

repository already exists at ...

Since (probably the last update) i get in the status mail a lot of lines like
A repository already exists at ssh://yunohost-backups@target-host/~/backup.
But borg list yunohost-backups@target-host:backup::auto_borg-2021-04-23_06:47 contains the backup (before the name did not have the leading "".
The corresponding journal log (sudo journalctl -u borg.service) reads

-- Logs begin at Fri 2021-04-23 01:23:26 CEST, end at Fri 2021-04-23 09:14:50 CEST. --
Apr 23 06:44:25 myyunohostdomain sudo[18021]: pam_unix(sudo:session): session closed for user root
Apr 23 06:44:26 myyunohostdomain sudo[17334]:     root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/yunohost app setting borg apps
Apr 23 06:44:26 myyunohostdomain sudo[17334]: pam_unix(sudo:session): session opened for user root by (uid=0)
Apr 23 06:46:31 myyunohostdomain sudo[17334]: pam_unix(sudo:session): session closed for user root
Apr 23 06:46:31 myyunohostdomain sudo[18323]:     root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/ls /etc/yunohost/apps/
Apr 23 06:46:31 myyunohostdomain sudo[18323]: pam_unix(sudo:session): session opened for user root by (uid=0)
Apr 23 06:46:31 myyunohostdomain sudo[18323]: pam_unix(sudo:session): session closed for user root
Apr 23 06:46:31 myyunohostdomain sudo[18330]:     root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/test ! -f /etc/yunohost/apps/borg/scripts/backup
Apr 23 06:46:31 myyunohostdomain sudo[18330]: pam_unix(sudo:session): session opened for user root by (uid=0)
Apr 23 06:46:32 myyunohostdomain sudo[18330]: pam_unix(sudo:session): session closed for user root
Apr 23 06:46:32 myyunohostdomain sudo[18335]:     root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/yunohost backup create -n auto_borg --method borg_app --apps borg
Apr 23 06:46:32 myyunohostdomain sudo[18335]: pam_unix(sudo:session): session opened for user root by (uid=0)
Apr 23 06:59:57 myyunohostdomain sudo[18335]: pam_unix(sudo:session): session closed for user root
Apr 23 06:59:57 myyunohostdomain sudo[20752]:     root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/test ! -f /etc/yunohost/apps/gitea/scripts/backup
Apr 23 06:59:57 myyunohostdomain sudo[20752]: pam_unix(sudo:session): session opened for user root by (uid=0)
Apr 23 06:59:57 myyunohostdomain sudo[20752]: pam_unix(sudo:session): session closed for user root
Apr 23 06:59:57 myyunohostdomain sudo[20755]:     root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/yunohost backup create -n auto_gitea --method borg_app --apps gitea
Apr 23 06:59:57 myyunohostdomain sudo[20755]: pam_unix(sudo:session): session opened for user root by (uid=0)
Apr 23 07:11:07 myyunohostdomain sudo[20755]: pam_unix(sudo:session): session closed for user root
Apr 23 07:11:07 myyunohostdomain sudo[23184]:     root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/test ! -f /etc/yunohost/apps/mautrix_telegram/scripts/backup
Apr 23 07:11:07 myyunohostdomain sudo[23184]: pam_unix(sudo:session): session opened for user root by (uid=0)
Apr 23 07:11:07 myyunohostdomain sudo[23184]: pam_unix(sudo:session): session closed for user root
Apr 23 07:11:08 myyunohostdomain sudo[23188]:     root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/yunohost backup create -n auto_mautrix_telegram --method borg_app --apps mautrix_telegram
Apr 23 07:11:08 myyunohostdomain sudo[23188]: pam_unix(sudo:session): session opened for user root by (uid=0)
Apr 23 07:13:42 myyunohostdomain sudo[24279]:     root : TTY=unknown ; PWD=/home/yunohost.backup/tmp/auto_mautrix_telegram/apps/mautrix_telegram/backup ; USER=postgres ; COMMAND=/bin/bash -c pg_dump mautrix_telegram
Apr 23 07:13:42 myyunohostdomain sudo[24279]: pam_unix(sudo:session): session opened for user postgres by (uid=0)
Apr 23 07:13:44 myyunohostdomain sudo[24279]: pam_unix(sudo:session): session closed for user postgres
Apr 23 07:17:51 myyunohostdomain sudo[23188]: pam_unix(sudo:session): session closed for user root
Apr 23 07:17:51 myyunohostdomain sudo[24955]:     root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/test ! -f /etc/yunohost/apps/monitorix/scripts/backup
Apr 23 07:17:51 myyunohostdomain sudo[24955]: pam_unix(sudo:session): session opened for user root by (uid=0)
Apr 23 07:17:51 myyunohostdomain sudo[24955]: pam_unix(sudo:session): session closed for user root
Apr 23 07:17:52 myyunohostdomain sudo[24957]:     root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/yunohost backup create -n auto_monitorix --method borg_app --apps monitorix
Apr 23 07:17:52 myyunohostdomain sudo[24957]: pam_unix(sudo:session): session opened for user root by (uid=0)
Apr 23 07:22:58 myyunohostdomain sudo[24957]: pam_unix(sudo:session): session closed for user root
Apr 23 07:22:58 myyunohostdomain sudo[26346]:     root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/test ! -f /etc/yunohost/apps/my_webapp/scripts/backup
Apr 23 07:22:58 myyunohostdomain sudo[26346]: pam_unix(sudo:session): session opened for user root by (uid=0)
Apr 23 07:22:58 myyunohostdomain sudo[26346]: pam_unix(sudo:session): session closed for user root
Apr 23 07:22:58 myyunohostdomain sudo[26349]:     root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/yunohost backup create -n auto_my_webapp --method borg_app --apps my_webapp
Apr 23 07:22:58 myyunohostdomain sudo[26349]: pam_unix(sudo:session): session opened for user root by (uid=0)
Apr 23 07:27:09 myyunohostdomain sudo[26349]: pam_unix(sudo:session): session closed for user root
Apr 23 07:27:09 myyunohostdomain sudo[27772]:     root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/test ! -f /etc/yunohost/apps/nextcloud/scripts/backup
Apr 23 07:27:09 myyunohostdomain sudo[27772]: pam_unix(sudo:session): session opened for user root by (uid=0)
Apr 23 07:27:09 myyunohostdomain sudo[27772]: pam_unix(sudo:session): session closed for user root
Apr 23 07:27:09 myyunohostdomain sudo[27779]:     root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/yunohost backup create -n auto_nextcloud --method borg_app --apps nextcloud
Apr 23 07:27:09 myyunohostdomain sudo[27779]: pam_unix(sudo:session): session opened for user root by (uid=0)
Apr 23 07:59:30 myyunohostdomain sudo[27779]: pam_unix(sudo:session): session closed for user root
Apr 23 07:59:30 myyunohostdomain sudo[580]:     root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/test ! -f /etc/yunohost/apps/synapse/scripts/backup
Apr 23 07:59:30 myyunohostdomain sudo[580]: pam_unix(sudo:session): session opened for user root by (uid=0)
Apr 23 07:59:30 myyunohostdomain sudo[580]: pam_unix(sudo:session): session closed for user root
Apr 23 07:59:31 myyunohostdomain sudo[583]:     root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/yunohost backup create -n auto_synapse --method borg_app --apps synapse
Apr 23 07:59:31 myyunohostdomain sudo[583]: pam_unix(sudo:session): session opened for user root by (uid=0)
Apr 23 07:59:49 myyunohostdomain sudo[1121]:     root : TTY=unknown ; PWD=/home/yunohost.backup/tmp/auto_synapse/apps/synapse/backup ; USER=postgres ; COMMAND=/bin/bash -c pg_dump matrix_synapse
Apr 23 07:59:49 myyunohostdomain sudo[1121]: pam_unix(sudo:session): session opened for user postgres by (uid=0)
Apr 23 08:01:44 myyunohostdomain sudo[1121]: pam_unix(sudo:session): session closed for user postgres
Apr 23 08:21:33 myyunohostdomain sudo[583]: pam_unix(sudo:session): session closed for user root
Apr 23 08:21:34 myyunohostdomain sudo[5894]:     root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/test ! -f /etc/yunohost/apps/ttrss/scripts/backup
Apr 23 08:21:34 myyunohostdomain sudo[5894]: pam_unix(sudo:session): session opened for user root by (uid=0)
Apr 23 08:21:34 myyunohostdomain sudo[5894]: pam_unix(sudo:session): session closed for user root
Apr 23 08:21:34 myyunohostdomain sudo[5897]:     root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/yunohost backup create -n auto_ttrss --method borg_app --apps ttrss
Apr 23 08:21:34 myyunohostdomain sudo[5897]: pam_unix(sudo:session): session opened for user root by (uid=0)
Apr 23 08:28:15 myyunohostdomain sudo[5897]: pam_unix(sudo:session): session closed for user root
Apr 23 08:28:16 myyunohostdomain sudo[7410]:     root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/test ! -f /etc/yunohost/apps/unattended_upgrades/scripts/backup
Apr 23 08:28:16 myyunohostdomain sudo[7410]: pam_unix(sudo:session): session opened for user root by (uid=0)
Apr 23 08:28:16 myyunohostdomain sudo[7410]: pam_unix(sudo:session): session closed for user root
Apr 23 08:28:16 myyunohostdomain sudo[7412]:     root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/yunohost backup create -n auto_unattended_upgrades --method borg_app --apps unattended_upgrades
Apr 23 08:28:16 myyunohostdomain sudo[7412]: pam_unix(sudo:session): session opened for user root by (uid=0)
Apr 23 08:35:36 myyunohostdomain sudo[7412]: pam_unix(sudo:session): session closed for user root
Apr 23 08:35:36 myyunohostdomain sudo[9009]:     root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/test ! -f /etc/yunohost/apps/wallabag2/scripts/backup
Apr 23 08:35:36 myyunohostdomain sudo[9009]: pam_unix(sudo:session): session opened for user root by (uid=0)
Apr 23 08:35:36 myyunohostdomain sudo[9009]: pam_unix(sudo:session): session closed for user root
Apr 23 08:35:36 myyunohostdomain sudo[9012]:     root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/yunohost backup create -n auto_wallabag2 --method borg_app --apps wallabag2
Apr 23 08:35:36 myyunohostdomain sudo[9012]: pam_unix(sudo:session): session opened for user root by (uid=0)
Apr 23 08:43:34 myyunohostdomain sudo[9012]: pam_unix(sudo:session): session closed for user root
Apr 23 08:43:35 myyunohostdomain sudo[11024]:     root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/yunohost app setting borg repository
Apr 23 08:43:35 myyunohostdomain sudo[11024]: pam_unix(sudo:session): session opened for user root by (uid=0)
Apr 23 08:43:41 myyunohostdomain sudo[11024]: pam_unix(sudo:session): session closed for user root
Apr 23 08:43:42 myyunohostdomain sudo[15379]: pam_unix(sudo:session): session closed for user root
Apr 23 08:43:42 myyunohostdomain systemd[1]: borg.service: Main process exited, code=exited, status=1/FAILURE
Apr 23 08:43:42 myyunohostdomain systemd[1]: borg.service: Failed with result 'exit-code'.
Apr 23 08:43:42 myyunohostdomain systemd[1]: Failed to start Run backup borg.

backup-with-borg automatically starts after server restart

Nov 19 11:22:27 ... systemd[1]: Started Run backup borg regularly.
Nov 19 11:22:30 ... systemd[1]: Starting Run backup borg...
then the backup-with-borg routine starts

I restarted to confirm this happens every time:
Nov 19 12:01:37 ... systemd[1]: Started Run backup borg regularly.
Nov 19 12:01:40 ... systemd[1]: Starting Run backup borg...
then the backup-with-borg routine starts

I have it set to backup daily. With two restarts the app will do 3 backups today.

Part of the passphrase is interpreted as an empty variable

Hello,
I installed Borg on my Yunohost servers and when I want to see if I can read backups, I can't because passphrase seems incorrect. So I dug into it.

The problem is, I think, in this script :

# cat /etc/yunohost/hooks.d/backup_method/05-borg_app
(...)
BORG_PASSPHRASE="RGjauGbbg6d$jj77g*piabfbfGedE"
(...)

(I changed the passphrase, obviously..)

Yunohost saved the complete passphrase :

# yunohost app setting borg passphrase
RGjauGbbg6d$jj77g*piabfbfGedE

But "$jj77g" is considered like an empty variable. Finally, borg used "RGjauGbbg6d*piabfbfGedE". We have to change " -> ' in the script.

$ PASSPHRASE="gaghtbgg$TEXT*gloa"
$ echo $PASSPHRASE 
gaghtbgg*gloa

$ PASSPHRASE='gaghtbgg$TEXT*gloa'
$ echo $PASSPHRASE 
gaghtbgg$TEXT*gloa

Perhaps we should add a warning during installation to say it's better to avoid some special characters ( ' , " , $ , .. ) ?

Support request: app keys / paths and other info.

Hello,
this is not an issue I think, but more a request for support, but some of this could be enhanced.

With yunohost app setting borg -h I get this output:

usage: yunohost app setting app key [-h] [-v VALUE] [-d]

Set or get an app setting value

positional arguments:
  app                   App ID
  key                   Key to get/set

optional arguments:
  -h, --help            show this help message and exit
  -v VALUE, --value VALUE
                        Value to set
  -d, --delete          Delete the key

But it seems to be impossible to get a list of the keys.

I am currently wondering where borg gets the host information from, meaning which is the path of the SSH-config file it uses. I would like to check why the backup seems to work as per journalctl output, but the remote repo does not contain current backups.

part of ls from remote repo:

drwxr-xr-x  1 user user    0 Nov 17 23:10 auto_borg_17_11_20_23:09/
drwxr-xr-x  1 user user    0 Nov 18 00:00 auto_borg_18_11_20_00:00/
drwxr-xr-x  1 user user    0 Nov 19 00:00 auto_borg_19_11_20_00:00/
drwxr-xr-x  1 user user    0 Nov 20 00:00 auto_borg_20_11_20_00:00/
drwxr-xr-x  1 user user    0 Nov 21 00:00 auto_borg_21_11_20_00:00/

Part of the journalctl output:

Mar 10 00:00:20 mail.example.com backup-with-borg[3832]: ls: cannot access '/etc/yunohost/hooks.d/backup/': No such file or directory
Mar 10 00:00:21 mail.example.com backup-with-borg[3832]: Creating a backup archive from the collected files...
Mar 10 00:00:22 mail.example.com backup-with-borg[3832]: Remote: Debian GNU/Linux 10
Mar 10 00:00:23 mail.example.com backup-with-borg[3832]: A repository already exists at ssh://[email protected]/~/backup.
Mar 10 00:01:38 mail.example.com backup-with-borg[3832]: Backup created
Mar 10 00:01:38 mail.example.com backup-with-borg[3832]: name: auto_data
Mar 10 00:01:38 mail.example.com backup-with-borg[3832]: results:
Mar 10 00:01:38 mail.example.com backup-with-borg[3832]:   apps:
Mar 10 00:01:38 mail.example.com backup-with-borg[3832]:   system:
Mar 10 00:01:38 mail.example.com backup-with-borg[3832]:     data_home: Success
Mar 10 00:01:38 mail.example.com backup-with-borg[3832]:     data_mail: Success

This looks good, but no idea where the backup goes to.

It seems that at some point borg switched the repo path from hostname:/volume1/shared/backup/some_name to hostname:~/backup/some_name, to the later one it does not have permission to write a backup, but it still seems it does to so so.

No I would like to look at all the config keys of the borg app and don't know how to figure out which keys are available.

New install not working

Hello!

I just got a new VPS for storing my backup, where I installed yunohost followed by borgbackup app.
Then, on my old yunohost server I just installed the borg app.

I followed the given instructions in the README.md file :-)

Then I type the following commands and retrieve an error:

# SRV=backupuser@mybackupserverip:~/backup
# BORG_RSH="ssh -i /root/.ssh/id_borg_ed25519 -oStrictHostKeyChecking=yes " borg list $SRV
Connection closed by remote host. Is borg working on the server?

Just to be sure, I have my public/private keys on my old server to backup, and I gave this public key while installing borgserver on my new server storing the backups.

I have absolutely no idea is that's a ssh or borg issue.

Léo

Don't consider backup as failed with an app without backup script

Describe the bug

The backup is considered as "failed" in the log mails when an app don't has any backup script :
Warning: The application ssh_chroot_dir__2 has no backup script. This app won't be backuped.

Expected behavior

Keep the warning in the log mails, but change the subject in "success" instead of "failed".

How to set it up in a simple manner?

Since ynh is meant to make thinks much easier, is Bog a simple backup solution?
I have a 1.4TB server with Debian. I have root access. I have ynh installed. Say I buy another server of 1.4TB, same setup. So, Server A and Server B. I install ynh on this server B and install ynh server. Now I setup the two. So that server A backups to server B, via that Borg and Borg Server panels in ynh.

So far I understood it correctly?

If so, how would a restore look like? Say I want to restore server A as it was a week ago? How is that going to be?

I am struggling to find an easy solution to backup my server. Thanks

Explanation on borg versioning backup

Hello,

Borg seems to work but when I do

sudo BORG_RSH="ssh -i /root/.ssh/id_borg_ed25519 -oStrictHostKeyChecking=yes " borg list ssh://server@serverb:portSSShserverb/~/backup

I get the following list

auto_conf_07_10_20_18:51             Wed, 2020-10-07 18:51:33 [xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx]
auto_data_07_10_20_18:51             Wed, 2020-10-07 18:51:52 [xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx]
auto_borg_07_10_20_19:24             Wed, 2020-10-07 19:24:10 [xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx]
auto_cesium_07_10_20_19:24           Wed, 2020-10-07 19:24:26 [xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx]
auto_element_07_10_20_19:24          Wed, 2020-10-07 19:24:47 [xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx]
auto_my_webapp_07_10_20_19:26        Wed, 2020-10-07 19:26:17 [xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx]
auto_nextcloud_07_10_20_19:27        Wed, 2020-10-07 19:27:23 [xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx]
auto_opensondage_07_10_20_21:15      Wed, 2020-10-07 21:15:17 [xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx]
auto_phpmyadmin_07_10_20_21:15       Wed, 2020-10-07 21:15:43 [xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx]
auto_searx_07_10_20_21:16            Wed, 2020-10-07 21:16:35 [xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx]
auto_synapse_07_10_20_21:17          Wed, 2020-10-07 21:17:24 [xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx]
auto_ttrss_07_10_20_22:01            Wed, 2020-10-07 22:01:53 [xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx]
auto_vpnclient_07_10_20_22:02        Wed, 2020-10-07 22:02:23 [xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx]
auto_wallabag2_07_10_20_22:02        Wed, 2020-10-07 22:02:46 [xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx]
auto_wordpress_07_10_20_22:03        Wed, 2020-10-07 22:03:35 [xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx]

it all date from 2020-10-07 (date I installed Borg) and I don't see any updated ARCHIVE_NAME.
Is it normal? or should I see more ARCHIVE_NAME or see them at least updated with a new date?
Or, if it's normal, how can I know it has been updated and it's not the old content only from the time I installed and launch bor backup?

Thank you in advance for your explanations.

borg fails due to bad app listing

yunohost app list --installed -b | grep id: | cut -d: -f2 gives errors on latest version of yunohost (specifically for the -b parameter)

should read
yunohost app list | grep id: | cut -d: -f2

Change backup time schedule

Hello,

I'm wondering how we can check and then change the backup time schedule?
I think I set it up to weekly but I would like to be sure and if I set it up to monthly then I'd like to change it.
How can we do that?

Regards,

borg cannot mount archives

Running borg_ynh on a raspberry pi all up-to-date, backups are successfully created. Now I'd like to examine the backup archives by fuse-mounting them at some place. In principle borg can do that, but the output of a borg mount command is presently:

borg mount not available: loading FUSE support failed [ImportError: No module named 'llfuse'

I tried adding llfuse with pip in borg's virtual environment but that fails:

/opt/borg-env/bin/python /opt/borg-env/bin/pip install llfuse
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Collecting llfuse
Downloading llfuse-1.4.0.tar.gz (478 kB)
|████████████████████████████████| 478 kB 4.4 MB/s
ERROR: Command errored out with exit status 2:
command: /opt/borg-env/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-d2mlx5w7/llfuse_974b8421bbac4f9490f2788fefef38f5/setup.py'"'"'; file='"'"'/tmp/pip-install-d2mlx5w7/llfuse_974b8421bbac4f9490f2788fefef38f5/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-_d4obr0o
cwd: /tmp/pip-install-d2mlx5w7/llfuse_974b8421bbac4f9490f2788fefef38f5/
Complete output (4 lines):
Package fuse was not found in the pkg-config search path.
Perhaps you should add the directory containing `fuse.pc'
to the PKG_CONFIG_PATH environment variable
No package 'fuse' found
----------------------------------------
ERROR: Command errored out with exit status 2: python setup.py egg_info Check the logs for full command output.

I tried adding python3-llfuse debian package and another borgbackup dependency according to debian, and recreating the python environment as in the install script, but that does not solve the issue.

However, if I install debian's borgbackup package and run the mount command with it (that is I invoke /usr/bin/borg instead of borg_yhn’s /usr/local/bin/borg -> /opt/borg-env/bin/borg) then mounting archives works fine. So it seems it's not an issue with a missing fuse kernel module as suggested in the Forum where I first reported the issue. Presently, debian is installing borg 1.1.9 while borg_ynh is pulling 1.1.13, which is maybe a clue. In any case, installing and using debian's borg for mounting the archives is a workaround for the moment.

I’m not familiar enough with python environments to understand what is going wrong and how to resolve that issue with borg_ynh.

_common.sh: line 14: virtualenv: command not found

Just trying to install borg client and have the same message on 4 différents server (some with YNH 3.4.2.4 and 3.5.2.2)

2019-06-16 10:35:24,078: WARNING - _common.sh: line 14: virtualenv: command not found
2019-06-16 10:35:24,080: DEBUG - + '[' '!' -d /opt/borg-env ']'
2019-06-16 10:35:24,081: WARNING - !!
2019-06-16 10:35:24,081: DEBUG - + virtualenv --python=python3 /opt/borg-env
2019-06-16 10:35:24,081: WARNING - borg's script has encountered an error. Its execution was cancelled.

Borg_ynh didn't send any email alert on backup error

Describe the bug

For some time the backup server was down, but I didn't receive any alert email that the backup failed.
I'm using the setting mailalert: errors_only.
I didn't have this problem some months ago, before the mailalert setting existed. Wondering if it's related, or not.

Context

  • Hardware: computer
  • YunoHost version: 4.2.8.3
  • I have access to my server: Through SSH, through the webadmin
  • Are you in a special context or did you perform some particular tweaking on your YunoHost instance?: no
  • Using, or trying to install package version/branch: 1.1.16~ynh24

Logs

These are the logs in /var/log/borg__2/211122_XXXX.err:

Failed to format translated string 'backup_applying_method_custom': 'Calling the custom backup method '{method}'...' with arguments '()' and '{}, raising error: KeyError('method') (don't panic this is just a warning)
Failed to format translatable string 'backup_applying_method_custom': 'Calling the custom backup method '{method}'...' with arguments '()' and '{}', raising  error: KeyError('method') (don't panic this is just a warning)
Connection closed by remote host. Is borg working on the server?
Could not run script: /etc/yunohost/hooks.d/backup_method/05-borg__2_app
Custom backup method could not get past the 'backup' step
Failed to format translated string 'backup_applying_method_custom': 'Calling the custom backup method '{method}'...' with arguments '()' and '{}, raising error: KeyError('method') (don't panic this is just a warning)
Failed to format translatable string 'backup_applying_method_custom': 'Calling the custom backup method '{method}'...' with arguments '()' and '{}', raising  error: KeyError('method') (don't panic this is just a warning)
Remote: ssh: connect to host backupServer.tld port 22: Network is unreachable
Connection closed by remote host. Is borg working on the server?
Could not run script: /etc/yunohost/hooks.d/backup_method/05-borg__2_app
Custom backup method could not get past the 'backup' step
Failed to format translated string 'backup_applying_method_custom': 'Calling the custom backup method '{method}'...' with arguments '()' and '{}, raising error: KeyError('method') (don't panic this is just a warning)
Failed to format translatable string 'backup_applying_method_custom': 'Calling the custom backup method '{method}'...' with arguments '()' and '{}', raising  error: KeyError('method') (don't panic this is just a warning)
Remote: ssh: connect to host backupServer.tld port 22: Network is unreachable
Connection closed by remote host. Is borg working on the server?
Could not run script: /etc/yunohost/hooks.d/backup_method/05-borg__2_app
Custom backup method could not get past the 'backup' step
Failed to format translated string 'backup_applying_method_custom': 'Calling the custom backup method '{method}'...' with arguments '()' and '{}, raising error: KeyError('method') (don't panic this is just a warning)
Failed to format translatable string 'backup_applying_method_custom': 'Calling the custom backup method '{method}'...' with arguments '()' and '{}', raising  error: KeyError('method') (don't panic this is just a warning)
Remote: ssh: connect to host backupServer.tld port 22: Network is unreachable
Connection closed by remote host. Is borg working on the server?
Could not run script: /etc/yunohost/hooks.d/backup_method/05-borg__2_app
Custom backup method could not get past the 'backup' step
Failed to format translated string 'backup_applying_method_custom': 'Calling the custom backup method '{method}'...' with arguments '()' and '{}, raising error: KeyError('method') (don't panic this is just a warning)
Failed to format translatable string 'backup_applying_method_custom': 'Calling the custom backup method '{method}'...' with arguments '()' and '{}', raising  error: KeyError('method') (don't panic this is just a warning)
Remote: ssh: connect to host backupServer.tld port 22: Network is unreachable
Connection closed by remote host. Is borg working on the server?
Could not run script: /etc/yunohost/hooks.d/backup_method/05-borg__2_app
Custom backup method could not get past the 'backup' step
Failed to format translated string 'backup_applying_method_custom': 'Calling the custom backup method '{method}'...' with arguments '()' and '{}, raising error: KeyError('method') (don't panic this is just a warning)
Failed to format translatable string 'backup_applying_method_custom': 'Calling the custom backup method '{method}'...' with arguments '()' and '{}', raising  error: KeyError('method') (don't panic this is just a warning)
Remote: ssh: connect to host backupServer.tld port 22: Network is unreachable
Connection closed by remote host. Is borg working on the server?
Could not run script: /etc/yunohost/hooks.d/backup_method/05-borg__2_app
Custom backup method could not get past the 'backup' step
Failed to format translated string 'backup_applying_method_custom': 'Calling the custom backup method '{method}'...' with arguments '()' and '{}, raising error: KeyError('method') (don't panic this is just a warning)
Failed to format translatable string 'backup_applying_method_custom': 'Calling the custom backup method '{method}'...' with arguments '()' and '{}', raising  error: KeyError('method') (don't panic this is just a warning)
Remote: ssh: connect to host backupServer.tld port 22: Network is unreachable
Connection closed by remote host. Is borg working on the server?
Could not run script: /etc/yunohost/hooks.d/backup_method/05-borg__2_app
Custom backup method could not get past the 'backup' step
Failed to format translated string 'backup_applying_method_custom': 'Calling the custom backup method '{method}'...' with arguments '()' and '{}, raising error: KeyError('method') (don't panic this is just a warning)
Failed to format translatable string 'backup_applying_method_custom': 'Calling the custom backup method '{method}'...' with arguments '()' and '{}', raising  error: KeyError('method') (don't panic this is just a warning)
Remote: ssh: connect to host backupServer.tld port 22: Network is unreachable
Connection closed by remote host. Is borg working on the server?
Could not run script: /etc/yunohost/hooks.d/backup_method/05-borg__2_app
Custom backup method could not get past the 'backup' step
Failed to format translated string 'backup_applying_method_custom': 'Calling the custom backup method '{method}'...' with arguments '()' and '{}, raising error: KeyError('method') (don't panic this is just a warning)
Failed to format translatable string 'backup_applying_method_custom': 'Calling the custom backup method '{method}'...' with arguments '()' and '{}', raising  error: KeyError('method') (don't panic this is just a warning)
Remote: ssh: connect to host backupServer.tld port 22: Network is unreachable
Connection closed by remote host. Is borg working on the server?
Could not run script: /etc/yunohost/hooks.d/backup_method/05-borg__2_app
Custom backup method could not get past the 'backup' step
It's hightly recommended to make your backup when the service is stopped. Please stop synapse service with this command before to run the backup 'systemctl stop matrix-synapse.service'
Failed to format translated string 'backup_applying_method_custom': 'Calling the custom backup method '{method}'...' with arguments '()' and '{}, raising error: KeyError('method') (don't panic this is just a warning)
Failed to format translatable string 'backup_applying_method_custom': 'Calling the custom backup method '{method}'...' with arguments '()' and '{}', raising  error: KeyError('method') (don't panic this is just a warning)
Connection closed by remote host. Is borg working on the server?
Could not run script: /etc/yunohost/hooks.d/backup_method/05-borg__2_app
Custom backup method could not get past the 'backup' step

These are the logs in /var/log/borg__2/211122_XXXX.log:

Creating a backup archive from the collected files...
The archive will contain about XX.XXGiB of data.
The operation 'Create a backup archive' could not be completed. Please share the full log of this operation using the command 'yunohost log share XXXXXXXX-XXXXXX-backup_create' to get help
Collecting files to be backed up for borg__2...
Loading installation settings...
Declaring files to be backed up...
Backup script completed for borg__2. (YunoHost will then actually copy those files to the archive).
Creating a backup archive from the collected files...
The archive will contain about XX.XXKiB of data.
The operation 'Create a backup archive' could not be completed. Please share the full log of this operation using the command 'yunohost log share XXXXXXXX-XXXXXX-backup_create' to get help
Collecting files to be backed up for APPNAME...
Loading installation settings...
Backing up the main app directory...
Backing up nginx web server configuration...
Backing up the PostgreSQL database...
Backing up fail2ban configuration
Backing up synapse configuration...
Backing up systemd configuration...
Backing up synapse data...
Backing up synapse log...
Backing up synapse hook...
Backup script completed for APPNAME. (YunoHost will then actually copy those files to the archive).
Creating a backup archive from the collected files...
The archive will contain about XX.XXGiB of data.
The operation 'Create a backup archive' could not be completed. Please share the full log of this operation using the command 'yunohost log share XXXXXXXX-XXXXXX-backup_create' to get help

doesn't work after upgrade from 1.1.10~ynh6 to 1.1.13~ynh2

When I upgrade the app from 1.1.10~ynh6 to 1.1.13~ynh2, it doesn't do automatic backup. I think that it does run, but somehow it doesn't create the backups on the specified server like before.
I am not using the corresponding backup app on the server that is used for storage but rather I manually configured the user and certificates, maybe something changed there?

Install failed because apt is not happy about the backports (because key is unknown)

During a test of this app, the install failed because :

W: GPG error: http://cdn-fastly.deb.debian.org/debian stretch-backports InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 8B48AD6246925553 NO_PUBKEY 7638D0442B90D010

was able to solve the issue by manually adding the key with :

gpg --keyserver pgpkeys.mit.edu --recv-keys 7638D0442B90D010
gpg --armor --export 7638D0442B90D010 | apt-key add -

(though the first one took 3 trials before succeeding on my machine ...?)

Backup of borg app fails with local storage

Borg configured to backup on a local storage.

Logs :

The backup miserably failed to backup borg application.
Source path '/root/.ssh/id_borg_ed25519' does not exist
Impossible de sauvegarder borg

So it also blocks update of the app :

L’opération 'Mettre à jour l’application 'borg'' a échoué ! Pour obtenir de l’aide, merci de partager le journal de l’opération en cliquant ici
Impossible de mettre à jour borg : Une erreur s’est produite durant l’exécution du script de mise à niveau de l’application
Backup failed, the upgrade process was aborted.

Add local backup

It would be great to be able to make local backups with borg, for instance on an external HDD.
Could it be integrated in this package ? Or is it better to create a dedicated app "borg local" ?

Failed to start Run backup borg

Describe the bug

Failed to start Run backup borg

borg.service: Failed with result 'exit-code'

A clear and concise description of what the bug is.

from log below

/usr/local/bin/backup-with-borg: line 83: syntax error near unexpected token `fi'

Context

  • Hardware: Raspberry Pi at home

  • YunoHost version: 4.2.7 (stable)

  • I have access to my server: Through SSH | through the webadmin

  • Are you in a special context or did you perform some particular tweaking on your YunoHost instance?: no

  • Using, or trying to install package version/branch:

  • If upgrading, current package version: version: 1.1.16~ynh21 (latest upgrade)

Steps to reproduce

  • upgrade borg from server
  • borg backup daily job worked fine untill latest upgrade
  • I've not changed settings after upgrading

Expected behavior

borg systemctl service executed without errors

Logs

Manually start borg service (I had seen borg failed in web interface)

systemctl start borg.service

borg.service - Run backup borg
Loaded: loaded (/etc/systemd/system/borg.service; static; vendor preset: enabled)
Active: failed (Result: exit-code) since Thu 2021-08-19 05:56:57 BST; 1min 16s ago
Process: 1581 ExecStart=/usr/bin/sudo /usr/local/bin/backup-with-borg borg (code=exited, status=2)
Main PID: 1581 (code=exited, status=2)

Aug 19 05:56:56 domain.tld sudo[5577]: pam_unix(sudo:session): session closed for user root
Aug 19 05:56:56 domain.tld sudo[5579]: root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/yunohost app setting borg mailalert
Aug 19 05:56:56 domain.tld sudo[5579]: pam_unix(sudo:session): session opened for user root by (uid=0)
Aug 19 05:56:57 domain.tld sudo[5579]: pam_unix(sudo:session): session closed for user root
Aug 19 05:56:57 domain.tld sudo[1581]: /usr/local/bin/backup-with-borg: line 83: syntax error near unexpected token `fi'

Aug 19 05:56:57 domain.tld sudo[1581]: /usr/local/bin/backup-with-borg: line 83: `fi'

Aug 19 05:56:57 domain.tld sudo[1581]: pam_unix(sudo:session): session closed for user root
Aug 19 05:56:57 domain.tld systemd[1]: borg.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Aug 19 05:56:57 domain.tld systemd[1]: borg.service: Failed with result 'exit-code'.
Aug 19 05:56:57 domain.tld systemd[1]: Failed to start Run backup borg.

Thanks.

Borg requesting user password

I have followed the instructions and I get into several issues. I was able to fix most of them, but this one, I can't find a solution.

I have setup borg on server A and borgserver on server B, both on YNH 4.X
BORG_RSH="ssh -i /root/.ssh/id_borg_ed25519 -oStrictHostKeyChecking=yes " borg list [email protected]:~/backup

I get a password prompt, which should not happen, since I'm using RSA keys.

I tried the same process with an empty test server A and the same server B, it seems the process worked flawlessly, I don't understand where is the issue. Note that I try the same command as above on this setup and I'm able to login in SSH without password.

Any ideas ?

prune options when several same applications delete backup

Hello,

I have several wordpress apps install on my server

When i check backup, at the end of it, i saw that some wordpress backup archive are missing

If you have more that 2 wordpress application (or other applications with the same name), when the prune hourly is execute (just after your backup), one of the archive is delete

For example
Backup works for

  • wordpress
  • wordpress__1
  • wordpress__2

But when the backup of the wordpress__2 is done the wordpress__1 is prune.

borg_ynh/conf/backup_method.j2

Line 47 in 07114d1

borg prune $repo -P ${name} --keep-hourly 2 --keep-daily=7 --keep-weekly=8 --keep-monthly=12 >> $LOGFILE 2>> $ERRFILE

For the moment i disable the prune, i don't know if it's a bug of borg which ignore the __ in the name of archive and how can i modify prune to ignore that

Send a mail if backup fails

It could be really helpful to be notified when a backup failed.
Some services send mails when succeed/failed

Password forget what is after a space

On initialisation, I set my passphrase to

Save of my server !

My last working save is on july 13
Since that, I have errors about passphrase
I tried

sudo yunohost app setting borg passphrase

And it only return

Save

I tried to restore it

sudo yunohost app setting borg passphrase -v "Save of my server !"

But the save still fails.
I tried a command suggested on the forum to get the actually used password

sudo grep "BORG_PASSPHRASE=" /etc/yunohost/hooks.d/backup_method/05-borg_app | cut -d'"' -f2

And it still returns

Save

So I edited the file, and the code only contains BORG_PASSPHRASE="Save"
I replaced it with the real password and now it works.

I don't know how, but my password was truncated.

There were a commit on July 14 containing

export passphrase=$(ynh_app_setting_get $app passphrase)

Maybe this needs some escaping to save the spaces ?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.