Volume missing after reboot about docker-volume-gluster HOT 35 OPEN

@trajano somethings don't work at least on travis but fuse is kind of tricky in this env (it failed to mount also from cli for exemple). I need to test it more locally.

from docker-volume-gluster.

The persistence is maintain via file /etc/docker-volumes/gluster/gluster-persistence.json in case of docker plugin this file is inside the container so if plugin is destroy the persistence is lost. I should mount this file on host.

from docker-volume-gluster.

Should be fix in https://github.com/sapk/docker-volume-gluster/releases/tag/v1.0.5

from docker-volume-gluster.

I guess this needs to be built with make ? I'm pretty terrible at running even the simplest of compile operations so I won't be able to test this, only waste your time. Looking at your other hub repo's I assume this cannot be automated with docker hub.

from docker-volume-gluster.

Simply upgrade the plugin: https://hub.docker.com/r/sapk/plugin-gluster/ https://docs.docker.com/engine/reference/commandline/plugin_upgrade/#examples
This will maybe erase current config of volume and you may have to re-create them again.

from docker-volume-gluster.

That is fine. The data exists elsewhere so recreating the volume only adds the ability for docker containers to mount that volume from the gluster server.

So after following the upgrade process, it failed.

root@localhost:/# docker plugin enable sapk/plugin-gluster

Error response from daemon: rpc error: code = Unknown desc = oci runtime error: container_linux.go:265: starting container process caused "process_linux.go:368: container init caused \"rootfs_linux.go:57: mounting \\\"/etc/docker-volumes/gluster\\\" to rootfs \\\"/var/lib/docker/plugins/dbfcaff7d50960238e085282706623ef279a1fc1c538dbdfb36a3985d066fecb/rootfs\\\" at \\\"/var/lib/docker/plugins/dbfcaff7d50960238e085282706623ef279a1fc1c538dbdfb36a3985d066fecb/rootfs/etc/docker-volumes/gluster\\\" caused \\\"no such device\\\"\""

from docker-volume-gluster.

I performed a forced remove of the plugin with docker plugin remove -f sapk/plugin-gluster and installed again with docker plugin install sapk/plugin-gluster and get the same error.

root@localhost:/# docker plugin install sapk/plugin-gluster
Plugin "sapk/plugin-gluster" is requesting the following privileges:
 - network: [host]
 - mount: [/etc/docker-volumes/gluster]
 - device: [/dev/fuse]
 - capabilities: [CAP_SYS_ADMIN]
Do you grant the above permissions? [y/N] y
latest: Pulling from sapk/plugin-gluster
a1a7381f86a6: Download complete
Digest: sha256:75d06c6afa4c0d82839710ce8bba781d66e8bb9fdbf6509ac7971ed4fa24bd31
Status: Downloaded newer image for sapk/plugin-gluster:latest 

Error response from daemon: rpc error: code = Unknown desc = oci runtime error: container_linux.go:265: starting container process caused "process_linux.go:368: container init caused \"rootfs_linux.go:57: mounting \\\"/etc/docker-volumes/gluster\ \\" to rootfs \\\"/var/lib/docker/plugins/8044bd305dd96e39425bb368479a3738e6f75edc1a80521315a617a429b5e5fb/rootfs\\\" at \\ \"/var/lib/docker/plugins/8044bd305dd96e39425bb368479a3738e6f75edc1a80521315a617a429b5e5fb/rootfs/etc/docker-volumes/gluste r\\\" caused \\\"no such device\\\"\""

from docker-volume-gluster.

Can you try to create folder /etc/docker-volumes/gluster on host ?

from docker-volume-gluster.

oh sorry I already did that after the first fail. I jus tried again after changing the owner of the folder from root to a normal user with the docker group. that didnt work either.

Error response from daemon: rpc error: code = Unknown desc = oci runtime error: container_linux.go:265: starting container process caused "process_linux.go:368: container init caused \"rootfs_linux.go:57: mounting \\\"/etc/docker-volumes/gluster\\\" to rootfs \\\"/var/lib/docker/plugins/8044bd305dd96e39425bb368479a3738e6f75edc1a80521315a617a429b5e5fb/rootfs\\\" at \\\"/var/lib/docker/plugins/8044bd305dd96e39425bb368479a3738e6f75edc1a80521315a617a429b5e5fb/rootfs/etc/docker-volumes/gluster\\\" caused \\\"no such device\\\"\""

from docker-volume-gluster.

root@localhost:/# ll /etc/docker-volumes/
bash: ll: command not found
root@localhost:/# ls -l /etc/docker-volumes/
total 4
drwxr-xr-x 2 appuser docker 4096 Oct 31 10:18 gluster

from docker-volume-gluster.

Ok I updated the dockerfile to create the folder also in the container. I was too optimistisc and chang without testing on my computer. I will try to test this futhermore soon.

from docker-volume-gluster.

The volume works again, but it is still missing after a reboot and I have to run docker volume create --driver sapk/plugin-gluster --opt voluri="gluster.mydomain.com:volume-app1" --name volume-app1 before starting the container. Is there some other flag I should be running to make the volume automatically restart?

from docker-volume-gluster.

I have restore the old version without the local mount of the config file. I will test it soon to list all the need changes and maybe build a plugin with a mountpoint (write the persitence file on host) and one with the file inside the container (like now) the let the user the choice.

from docker-volume-gluster.

After reviewing code, I found that the reading and saving part of persistence wasn't using the same file.
Should be really fix in this last version. For information : v1.0.5...master#diff-c2830bc69fcc3e78c6bd782f1a5d8920L56 vs

from docker-volume-gluster.

the problem still occurs.
You need to remove and re-add the plugin.

    "Id": "5fbb180f3e90158bbc74f459383131890d2a02d0c90e4944e60445fde3712675",
    "Name": "sapk/plugin-gluster:latest",
    "PluginReference": "docker.io/sapk/plugin-gluster:latest",

from docker-volume-gluster.

from docker-volume-gluster.

from docker-volume-gluster.

Still broken. After a host reboot, the volume is completely borked. It shows in docker volume ls but cannot be removed and is empty when mounted by the . I removed all containers and ran docker volume rm volume-openvpn and got the error message Error response from daemon: unable to remove volume: remove volume-openvpn: VolumeDriver.Remove: volume volume-openvpn is currently used by a container even though no containers exist on the host. I removed all images and get the same error removing the volume. Removing the plugin with docker plugin rm sapk/plugin-gluster shows the error Error response from daemon: plugin sapk/plugin-gluster:latest is in use which is expected after the first error. The plugin can be removed forcefully with

$ docker plugin rm -f sapk/plugin-gluster
sapk/plugin-gluster
$ docker plugin ls
ID                  NAME                DESCRIPTION         ENABLED
$_

The plugin and volume are gone at this point. I can then install the plugin, create the volume, and pull/run the container again and everything is functional. I then wait 5 minutes and reboot the host.

The host comes back up and the volume is empty, container is continually restarting. I remove the container and image and cannot remove the volume.

from docker-volume-gluster.

It seems that the plugin doesn't take in account the removed container attached that why it block the removal of volume. For information, a volume plugin doesn't know the running container and can only keep count of request. I will look if it is now possible to pass arg to plugin to force removal. For the original problem, I think that docker (re-)start the container before the plugin is ready. Do you start the container from systemd/init.d or let docker restart running container before reboot ?

from docker-volume-gluster.

from docker-volume-gluster.

Seems related to #12

from docker-volume-gluster.

from docker-volume-gluster.

@cron410 I have rework the plugin to have a common base for the multiple docker volume I have developed. I hope it fix your issue.

from docker-volume-gluster.

from docker-volume-gluster.

Hi I think something else seems off. I am putting it here in case what I write can be the same test case for this issue.

I am not getting any errors on the plugin (even without the subdir). I see it in docker volumes ls in the swarm node

However, when I actually try to mount the volume using mount -t glusterfs from another system i do not see the files. However the volume is still there and I can ls within the docker.

Perhaps we should have a unit test to include doing a mount of the glusterfs and see if the file is there.

from docker-volume-gluster.

@trajano there is a test doing that https://github.com/sapk/docker-volume-gluster/blob/master/gluster/integration/integration_test.go. It create a cluster of gluster node (container) and try to mount them via managed and legacy plugin. It create folder, write to it and compare value of data inside.

from docker-volume-gluster.

from docker-volume-gluster.

from docker-volume-gluster.

I saw your recent commits added the explicit mount. Did it work for you?

from docker-volume-gluster.

Tried a few more things. The examples with docker-compose and docker run work. However, it has issues when I deploy it on the swarm. On Swarm mode it looks like it is just running a local volume.

from docker-volume-gluster.

Okay I thought it worked since the travis build on the branch passed. I'm doing a few more tests on my side too.

from docker-volume-gluster.

Tried to do a reboot again. This time with the legacy plugin (rather than managed) seems to have the same behavior in that if the remote store is not ready (which may take a minute or so) it creates a local volume rather than go on a loop.

I can verify that it is still using the gluster mount in the service configuration when I did a docker service inspect

  "Mounts": [
                        {
                            "Type": "volume",
                            "Source": "ci_jenkins_home",
                            "Target": "/var/jenkins_home",
                            "VolumeOptions": {
                                "Labels": {
                                    "com.docker.stack.namespace": "ci"
                                },
                                "DriverConfig": {
                                    "Name": "gluster",
                                    "Options": {
                                        "voluri": "store1,store2:trajano/jenkins"
                                    }
                                }
                            }
                        }
                    ],

from docker-volume-gluster.

I took a crack at making my own plugin, so far so good and it seems to survive reboots but only in docker stack deployed nodes. https://hub.docker.com/r/trajano/glusterfs-volume-plugin/ It does not sustain itself when the volume is created using docker volume create (also applies to docker-compose up) and then reboot.

from docker-volume-gluster.

I think I solved it on mine by storing the volume map data to a bolt db that is inside the rootfs and I also used "global" scope.

from docker-volume-gluster.

from docker-volume-gluster.