Explore 3rd party alternatives to providing our own vagrant k8s deployment about gluster-kubernetes HOT 7 OPEN

@ravishivt @v1k0d3n @intlabs ....well hot dang! :D I'll try to get playing with this next week. :)

from gluster-kubernetes.

That you for the feedback and suggestions! I'll try to address them here as best I can. :)

1. Pardon my ignorance, I imagine you're talking about installing packages on the k8s nodes to actually be able to mount GlusterFS volumes, yes? If so, what does the glusterfs-client package contain?
2. I would certainly love to have this, I'm just restricted by time. :) It would require some careful restructuring of the ansible playbooks, but I know it can be done. If you or someone you know would be willing to help out with this, I would ask to have a separate issue raised for it and we can have the discussion there.
3. Currently we don't run the deploy script as part of the ansible provisioning because we want to have the user walk through the deployment themselves. The scope of the project currently isn't in bringing up a full cluster stack form scratch, it's specifically on the deployment of a hyper-converged GlusterFS setup on Kubernetes. The vagrant-ansible setup is a PoC/development demo, and not something we would recommend for production deployment. I would also think that, while it would provide many logical conveniences, moving to a pure ansible implementation would limit the portability of the deployment.
   All THAT said, something we could do is automatically copy the deploy directory into the master node of the vagrant setup so all you'd have to do is log in, edit the topology file (I don't want to hide the importance of needing to understand the topology file), and run the script. Would that be a useful compromise?

from gluster-kubernetes.

1. Yup, the glusterfs-client package allows Ubuntu to mount "glusterfs" filesystem types.
2. I think the ansible playbook that targets the specific Vagrantfile setup isn't necessary if we go with a general playbook (see part 3). Actually, it might be better to refer users to a project that is dedicated to getting a kubernetes or openshift deployment up and have this project focus on getting the GlusterFS+Heketi hyper-converged deployment up on an existing k8s/OCP deployment. For a k8s setup referral, I'd suggest https://github.com/att-comdev/halcyon-kubernetes. Alternatively, we could git submodule the other project(s).
3. I agree that this project shouldn't be focused on bringing up a full cluster from scratch, see my comment in part 2 but I don't see a reason why we have to make the user manually go through all the steps. The user would still need to specify a topology.json file and an ansible inventory file but then the ansible role would then do all the manual stuff for them. I've gotten this set up in my fork, https://github.com/ravishivt/gluster-kubernetes/tree/myenv/deploy. You specify a k8s cluster that's already up and running in inventory.ini and topology.json and then run:
   ansible-playbook -i inventory.ini -e 'ansible_user=deploy' deploy.yml
   The ansible roles do everything for you including:

• upgrade-os: Upgrade OS packages.
• build-heketi: Builds heketi from source. This may not be necessary when heketi posts latest release binaries, see heketi/heketi#627, although it is still useful to build for targets with uncommon architectures.
• gk-prep: Installs glusterfs-client 3.8 and loads required kernel modules.
• gk-deploy: Copies the gk-deploy script, k8s/ocp templates, topology, and then runs the gk-deploy script.
• gk-post: Automatically finds the HEKETI_SERVER_CLI and creates the StorageClass under a user-configured name from group_vars/all.yml.

The playbook has a few empty placeholders for CentOS and OpenShift specific logic but it's working well for my Ubuntu k8s cluster.

There's also an abort.yml playbook to delete the StorageClass and run gk-deploy with --abort. Lastly, there's a destroy_vg.yml playbook to find Volume Groups and delete them after a confirmation prompt.

from gluster-kubernetes.

Hi! Apologies for the delay, your last comment hit around when I was heading on end-of-year vacation.

Hmm... not bad! Thus far there seems there are two issues at play here:

1. Establish a convenient, more generic procedure for bringing up a development environment with Kubernetes already running and ready to go.
2. Use Ansible to help automate the pre-requisites, running, and removal of the deployment.

As you just submitted #149 for item 2, I will address item 1. I also suggest editing the subject of this Issue to reflect this. :)

To be frank, the entire vagrant/ directory started as just a development convenience for me. If it can be replaced with a more general solution using git submodules I'm all for it. I had a look at the halcyon-kubernetes project and I find it lacking since it doesn't support Fedora or libvirt (Red Hat, sorry ;) ). I wouldn't be able to get a good look at it and modify it for my use for some time, I think, but if we could I agree this would make the project a lot more accessible.

from gluster-kubernetes.

No need to apologize, thanks for getting back to me. I updated this issue's title as you suggested.

Take a look at https://github.com/att-comdev/halcyon-vagrant-kubernetes. It is a separate project that submodules halcyon-kubernetes. This project uses vagrant to stand up X nodes using VirtualBox, libvrt, or OpenStack. Once set up, it then runs the ansible playbook from halcyon-kubernetes to initialize the cluster on those nodes. That being said, Fedora is not officially supported as you pointed out.

The difficulty is adding the vagrant logic to add virtual disk(s) to all those environments: VirtualBox, libvrt, and OpenStack. I used similar vagrant code as what is in this project but it never worked consistently. Eventually, I gave up and manually set up 3x VMware instances.

from gluster-kubernetes.

@ravishivt, i just happened to notice the issues/PR you created on our halcyon-vagrant-kubernetes repo, and we'd be happy to support fedora if it helps your needs. @intlabs and a larger kolla-kubernetes are using the project, so we're dedicated to maintaining and improving it. i will be able to read through, review and address your items tomorrow. thanks for the interest and PR!

Also, @jarrpa, project does support KVM, but if you find there are issues with it (or require libvirt) we'd be happy to improve on this functionality. https://github.com/att-comdev/halcyon-vagrant-kubernetes/blob/master/Vagrantfile#L82-L87

from gluster-kubernetes.

@jarrpa Ironically I only use halcyon-kubernetes with CentOS/Fedora hosts and Libvirt. I need to get the documentation in the GitHub repo updated, but in the mean-time, you may find the documents I wrote for Kolla-Kubernetes useful, http://docs.openstack.org/developer/kolla-kubernetes/development-environment.html. Hope this helps.

I've been using Halcyon for Ceph, and so it would be great to get Gluster running as well. If there's any way I can help with that just give me a shout :)

@ravishivt Thanks for your PRs, they were great additions. I must confess I have only been using the playbooks in the Vagrant environment, and so had not tested with multiple NICs. Cheers!

from gluster-kubernetes.