nextcloud.export not exporting userdata about nextcloud-snap HOT 17 OPEN

Hallo @scubamuc

As I managed to test, the issue could be on nextcloud.export making a backup on the default datadirectory /var/snap/nextcloud/common/nextcloud/data and not the one I set in /var/snap/nextcloud/41512/nextcloud/config/config.php. This directory is currently empty on my server, but when I create a file inside this directory (a .mp4 file with ~118MB) the nextcloud.export command gives this output:

WARNING: This functionality is still experimental and under
development, use at your own risk. Note that the CLI interface is
unstable, so beware if using from within scripts.

Enabling maintenance mode... done
Exporting apps...
        863.38M 100%   26.39MB/s    0:00:31 (xfr#5636, to-chk=0/6476)
Exporting database...
Exporting config...
Exporting data...
        123.61M 100%   85.55MB/s    0:00:01 (xfr#3, to-chk=0/4)

Successfully exported /var/snap/nextcloud/common/backups/20240423-105044
Disabling maintenance mode... done

du -hs /var/snap/nextcloud/common/nextcloud/data/ will give me the following output:
118M /var/snap/nextcloud/common/nextcloud/data/

If I now remove anything inside this directory again (directory size: 8,0K /var/snap/nextcloud/common/nextcloud/data/) I will get this output again with nextcloud.export:

WARNING: This functionality is still experimental and under
development, use at your own risk. Note that the CLI interface is
unstable, so beware if using from within scripts.

Enabling maintenance mode... done
Exporting apps...
        863.38M 100%   30.53MB/s    0:00:26 (xfr#5636, to-chk=0/6476)
Exporting database...
Exporting config...
Exporting data...
            542 100%    0.00kB/s    0:00:00 (xfr#2, to-chk=0/3)

Successfully exported /var/snap/nextcloud/common/backups/20240423-105540
Disabling maintenance mode... done

I will use those workarounds and scripts to backup my data at the moment, thank you for the provided links.
I hope I was able to help to find the root of this issue. Mabye it was between the last one or two releases and I didn't notice.

from nextcloud-snap.

I think @Albatross29 is correct, this is a separate feature request.

from nextcloud-snap.

@scubamuc
Sorry for my new comment.
I have to correct what I said in the message before.
Now I have created a new data directory on another drive in my vm and changed this directory:
'datadirectory' => '/mnt/cloud/data'
When I run nextcloud.export now this is the output I get:

WARNING: This functionality is still experimental and under
development, use at your own risk. Note that the CLI interface is
unstable, so beware if using from within scripts.

Enabling maintenance mode... done
Exporting apps...
              0 100%    0.00kB/s    0:00:00 (xfr#0, to-chk=0/1)
Exporting database...
Exporting config...
Exporting data...
          3.84M 100%  261.80MB/s    0:00:00 (xfr#41, to-chk=0/128)

Successfully exported /var/snap/nextcloud/common/backups/20240423-100000
Disabling maintenance mode... done

Seems like when running nextcloud.export it will try to export the data from the default directory /var/snap/nextcloud/common/nextcloud/data and not the one I set in the config.php file.

Can you verify that on your vanilla instance?

These are the steps I did to create my VM:

# Fully updating the system
sudo apt update && sudo apt upgrade -y

# Formatted the other drive
sudo parted /dev/sdb mklabel gpt
sudo parted -a opt /dev/sdb mkpart primary ext4 0% 100%
sudo mkfs.ext4 -L partitionsname /dev/sdb1

sudo mkdir /mnt/cloud
sudo nano /etc/fstab
# Pasted the following line in /etc/fstab
/dev/sdb1   	/mnt/cloud  ext4    defaults    0   2

sudo mount -a

# Opening the ip in browser and finish the installation (user admin, pw 123456)
sudo cp -a /var/snap/nextcloud/common/nextcloud/data/. /mnt/cloud/data
sudo nano /var/snap/nextcloud/41512/nextcloud/config/config.php
# Changed 'datadirectory' to /mnt/cloud/data

sudo snap disable nextcloud
sudo snap enable nextcloud
sudo snap connect nextcloud:removable-media

Sorry for any inconvenience.

from nextcloud-snap.

Servus @Albatross29,

no inconvenience whatsoever... glad we got to the bottom of this.

yes this seems to be an issue with data in an alternative data directory see #2057.

check out the alternative backup, restore & migrate

from nextcloud-snap.

This is a shortcoming in the script. Ideally it would check the Nextcloud config and use the data directory configured there, but it doesn't do that today.

from nextcloud-snap.

I would say my case is a bit different because #2057 asks to specify the backup directory if wanted, but for me it's not even possible to create backups with the data being stored outside the default datadirectory.

from nextcloud-snap.

welcome @Albatross29 and thank you for your request

could you show us your directory sudo ls -al /var/snap/nextcloud/common/backups/20240421-204532

it should look something like this;

you should see the data directory. In there should be your user directory;

from nextcloud-snap.

Hello @scubamuc and thank you for taking your time to help me with my problem!
Here is the output of sudo ls -al /var/snap/nextcloud/common/backups/20240422-195111:

total 5744
drwxr-x---  4 root root    4096 Apr 22 19:51 .
drwxr-xr-x  4 root root    4096 Apr 22 19:51 ..
drwxr-x--- 18 root root    4096 Apr 21 20:21 apps
-rw-r--r--  1 root root    1445 Apr 22 19:51 config.php
drwxrwxr-x  2 root root    4096 Dez 25 20:29 data
-rw-r--r--  1 root root 5856578 Apr 22 19:51 database.sql
-rw-r--r--  1 root root       2 Apr 22 19:51 format

Output of sudo ls -al /var/snap/nextcloud/common/backups/20240422-195111/data:

total 12
drwxrwxr-x 2 root root 4096 Dez 25 20:29 .
drwxr-x--- 4 root root 4096 Apr 22 19:51 ..
-rw-rw-r-- 1 root root  542 Dez 22 13:12 .htaccess
-rw-rw-r-- 1 root root    0 Dez 22 13:12 .ocdata

Please tell me if you need more information!

Kind regards,
Leo

from nextcloud-snap.

@Albatross29, not knowing what hardware you're using, I'm thinking along the lines that you're out of space. which is why nextcloud.export is not able to copy your data. Your data is going to be the largest chunk in the backup. nextcloud.export is sweet and simple and not very talkative... so its a guessing game. There's certainly nothing wrong with the script though, it worked perfectly on the instance I created to reproduce your issue ... see screen shots. so there must be some other issue.

• Is your nextcloud.export backup scripted or automated (beware of using within scripts)
• Where are you copying/moving the backups to?
• Is there enough space where the backups are being copied/moved to?
• You could try a rotating backup see export & import
• Or you could do it the snap way with rotating snap snapshots... zero downtime and easy migration see backup, restore & migrate

from nextcloud-snap.

@scubamuc I created the backup directory via symlink according to this tutorial.
Output of ll /var/snap/nextcloud/common/:

total 12
drwxr-xr-x 3 root root 4096 Apr 23 04:34 ./
drwxr-xr-x 5 root root 4096 Apr 21 21:27 ../
lrwxrwxrwx 1 root root   25 Apr 23 04:34 backups -> /mnt/nfs/nextcloud_backup/
drwxr-x--- 4 root root 4096 Dez 25 20:25 nextcloud/

My data directory is not at the default path, but on another partition and mounted on /mnt/cloud/data (also correctly set in config.php).

The output of du -hs /mnt/cloud/data:
14G /mnt/cloud/data

Output of df -h:

Filesystem                         Size  Used Avail Use% Mounted on
tmpfs                              1,4G  1,9M  1,4G   1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv   54G   25G   26G  49% /
tmpfs                              6,8G  4,0K  6,8G   1% /dev/shm
tmpfs                              5,0M     0  5,0M   0% /run/lock
/dev/sdd2                          2,0G  253M  1,6G  14% /boot
/dev/sdd1                          1,1G  6,1M  1,1G   1% /boot/efi
/dev/md0p1                         3,6T   14G  3,4T   1% /mnt/cloud
/dev/md5p1                         7,3T  1,4T  5,5T  21% /mnt/plex
10.0.0.3:/jswiki_backup             33T  5,0T   28T  16% /mnt/nfs/jswiki_backup
10.0.0.3:/nextcloud_backup          33T  5,0T   28T  16% /mnt/nfs/nextcloud_backup
overlay                             54G   25G   26G  49% /var/lib/docker/overlay2/6b32bf18744f53459fc152526978ac748a0145f3254af74f0b9e369b9ba970eb/merged
overlay                             54G   25G   26G  49% /var/lib/docker/overlay2/97a5f73873d14890db26ff983ebed4e28d5a72e156eecfa835d4b312ddd67fe0/merged
overlay                             54G   25G   26G  49% /var/lib/docker/overlay2/c7a7a4062d53b5d5ff4e7f272851e08a42a1a87c19d8627ef7e0ab2a66f572c3/merged
tmpfs                              1,4G  4,0K  1,4G   1% /run/user/1000

To answer your questions:

• I normally run it via cronjob everyday, but the behaviour is the same whether executed via cronjob or manually
• nextcloud.export is correctly moving the backup to /var/snap/nextcloud/common/backups, which is a symlink to /mnt/nfs/nextcloud_backup
• I have enough free space for the backups

from nextcloud-snap.

@Albatross29, oh I see the root cause now...

according to this tutorial.

this chap is excluding data in his command in that tutorial

nextcloud.export -abc

copy and pasting commands without understanding is a bad idea...

you obviously need data copied too, so you may want to use nextcloud.export -abcd or nextcloud.export without options as per docs. see the docs nextcloud.export --help and consult the wiki

so this is certainly not a snap issue nor is it a nextcloud.export issue.

from nextcloud-snap.

@scubamuc I never said I excluded my data directory. I only followed the tutorial to create the symlink, not exporting data.

If I wouldn't run the full nextcloud.export command it wouldn't even show the data topic in the command line.
Output of running nextcloud.export -abc:

WARNING: This functionality is still experimental and under
development, use at your own risk. Note that the CLI interface is
unstable, so beware if using from within scripts.

Enabling maintenance mode... done
Exporting apps...
        863.38M 100%   24.24MB/s    0:00:33 (xfr#5636, to-chk=0/6476)
Exporting database...
Exporting config...

Successfully exported /var/snap/nextcloud/common/backups/20240423-074529
Disabling maintenance mode... done

The data category is missing here.

I've been running nextcloud.export as a cronjob for many months and it worked until a few days or so (I just realised recently it's not working anymore).
Besides that, you can see in my messages I always referenced to the full command nextcloud.export

from nextcloud-snap.

@Albatross29,

you can see in my messages I always referenced to the full command nextcloud.export

yes that's true. I jumped the gun... and made an idiot of myself for patronising you... sorry for that

so we've tried reproducing the issue on a fresh instance, checked space and checked typos in the nextcloud.export command,

the only other cause could be your symlinked directory. please check permissions of that symlink and double check mounting removable media

I've been running nextcloud.export as a cronjob for many months and it worked until a few days or so (I just realised recently it's not working anymore)

can you recall what changed between the last successful and now?

from nextcloud-snap.

@scubamuc

Don't be harsh on yourself. I appreciate your words, all good for me! Thank you.

I sadly can't tell what exactly changed and when it happened as I only keep 7 backups...
My NAS OS got updated and my Ubuntu Server itself. That's it afaik (beside Nextcloud Snap getting updated with snap refresh)

I checked if the backup will be correctly exported in the default directory (deleting /var/snap/nextcloud/common/backups so symlink will be removed and ran nextcloud.export -d).
Output:

WARNING: This functionality is still experimental and under
development, use at your own risk. Note that the CLI interface is
unstable, so beware if using from within scripts.

Enabling maintenance mode... done
Exporting data...
            542 100%    0.00kB/s    0:00:00 (xfr#2, to-chk=0/3)

Successfully exported /var/snap/nextcloud/common/backups/20240423-083218
Disabling maintenance mode... done

Output of ll /var/snap/nextcloud/common/:

total 16
drwxr-xr-x 4 root root 4096 Apr 23 08:32 ./
drwxr-xr-x 5 root root 4096 Apr 23 08:32 ../
drwxr-xr-x 3 root root 4096 Apr 23 08:32 backups/
drwxr-x--- 4 root root 4096 Dez 25 20:25 nextcloud/

The NAS permissions seem to be correct to me - I can read and write any files or directories.
I tried the following steps:

• Created symlink directory sudo ln -s /mnt/nfs/nextcloud_backup /var/snap/nextcloud/common/backups
• Ran sudo snap connect nextcloud:removable-media
  Output:

WARNING: This functionality is still experimental and under
development, use at your own risk. Note that the CLI interface is
unstable, so beware if using from within scripts.

Enabling maintenance mode... done
Exporting data...
            542 100%    0.00kB/s    0:00:00 (xfr#2, to-chk=0/3)

Successfully exported /var/snap/nextcloud/common/backups/20240423-090405
Disabling maintenance mode... done

Output of ls -al /var/snap/nextcloud/common/backups/20240423-090405/

total 16
drwxr-s---  3 root root 4096 Apr 23 09:04 .
drwxrwsrwx 19 root root 4096 Apr 23 09:04 ..
drwxrwxr-x  2 root root 4096 Dez 25 20:29 data
-rw-rw-rw-  1 root root    2 Apr 23 09:04 format

The data directory inside is empty.

from nextcloud-snap.

@scubamuc
I tried the following procedure in a VM on my computer:

# Fully updating the system
sudo apt update && sudo apt upgrade -y
# Installing nextcloud via snap
sudo snap install nextcloud
# Opening the ip in browser to finish installation

Now, nextcloud.export gives me the following output:

WARNING: This functionality is still experimental and under
development, use at your own risk. Note that the CLI interface is
unstable, so beware if using from within scripts.

Enabling maintenance mode... done
Exporting apps...
              0 100%    0.00kB/s    0:00:00 (xfr#0, to-chk=0/1)
Exporting database...
Exporting config...
Exporting data...
          2.20G 100%  224.97MB/s    0:00:09 (xfr#90, to-chk=0/180)

Successfully exported /var/snap/nextcloud/common/backups/20240423-093811
Disabling maintenance mode... done

It seems like the issue must be inside my infrastructure.
I have to investigate further into this, but as you mentioned, doesn't look like the issue is on nextcloud or snap itself.

Thank you for your help and patience!

from nextcloud-snap.

@kyrofa is this configuarable?

export-data

export_data()
{
	backup="$1"
	echo "Exporting data..."
	if ! rsync -ah --info=progress2 "${NEXTCLOUD_DATA_DIR%/}/" "${backup}/data"; then
		echo "Unable to export data"
		exit 1
	fi
}

from nextcloud-snap.

I guess we'll close this, since its already tagged as feature request in #2057

from nextcloud-snap.