deploy-heketi stuck on "Adding device" about gluster-kubernetes HOT 6 CLOSED

It looks like you re-ran the script after it already formatted the virtual disks. You have to either destroy and recreate the vagrant setup or go to the nodes and manually delete the VGs on each device.

from gluster-kubernetes.

Here's a brand new deploy with the same issue. It gets stuck on adding a device to the second node.

ubuntu@kube1:~/deploy$ export KUBECONFIG="/etc/kubernetes/admin.conf" && sudo -E bash ./gk-deploy -g -w 180
Using Kubernetes CLI.
serviceaccount "heketi-service-account" created
node "172.16.35.11" labeled
node "172.16.35.12" labeled
node "172.16.35.13" labeled
daemonset "glusterfs" created
Waiting for GlusterFS pods to start ... OK
service "deploy-heketi" created
deployment "deploy-heketi" created
Waiting for deploy-heketi pod to start ... OK
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    17  100    17    0     0   2549      0 --:--:-- --:--:-- --:--:--  2833
Creating cluster ... ID: 58fd7a026bf01bd88ef9d3101a7b8911
        Creating node 172.16.35.11 ... ID: 74fd6bcbb1b3f207d5c84a196ea045eb
                Adding device /dev/sdb ... OK
                Adding device /dev/sdc ... OK
                Adding device /dev/sda ... OK
        Creating node 172.16.35.12 ... ID: 2a3b8b375e1063eb9f888190488092ba
                Adding device /dev/sdb ...

from gluster-kubernetes.

Hmm... my off-hand guess, to kind of state the obvious: this is either a bug in the VM setup or in heketi. I'd say start by editing the gk-deploy script to insert an exit statement just after https://github.com/gluster/gluster-kubernetes/blob/master/deploy/gk-deploy#L382 . Then, check the output of kubectl nodes; kubectl get pod,sa to make sure all the nodes are Ready and the GlusterFS and deploy-heketi pods are Running. kubectl describe the deploy-heketi pod and check that it has no events, and verify there are no error messages in the logs. Then, run heketi-cli -s http://${heketi_service} topology load --json=${TOPOLOGY} (with the variables fileld in or set, of course) and check the logs again to see if this comes up with anything useful.

If you find something in those logs you can't immediately track down, raise an issue on https://github.com/heketi/heketi and we'll work to track it down there, referencing this issue.

@MohamedAshiqrh or @humblec, can you help by pointing out where to find logs for the heketi pod? I'm still fuzzy on that.

from gluster-kubernetes.

Thanks @jarrpa, good suggestions. I wasn't able to pinpoint the error from the logs or events but I do see it looping on a GET /queue/ call. I raised on issue on heketi at heketi/heketi#625 with all my logs. We can close this issue if you'd like.

from gluster-kubernetes.

I had this issue and it turned out to be an extra space in my topology.json file which made the hostname invalid. Double check your topology file!

from gluster-kubernetes.

Somehow I missed the latest update! Thanks @johnfosborneiii. :) Closing this issue.

from gluster-kubernetes.