Helm? about kube-airflow HOT 19 CLOSED

@andrewrothstein I'm working on that a bit as well. The addition of RBAC roles in 1.6 has thrown us for a loop and helm charts are really helpful.

Are you planning to fork this repo or create a separate one for the helm chart?

from kube-airflow.

Your call. Would ultimately like to get it merged into https://github.com/kubernetes/charts.

from kube-airflow.

I would agree. I'm working on the configmap piece now and should have something relatively usable in the next week or so. Still a bit new to the whole helm thing so it's taking a bit longer.

from kube-airflow.

I think I'm going to do this with a gitvolume for the dags. Does that seem like a good idea? Have other suggestions? We could always implement other dag backends for NFS, AWS, etc, but I'd like to start somewhere.

from kube-airflow.

I'm airflow novice so bear with me. The scheduling of DAGs would seem to be a separate concern from launching the infrastructure / processes that constitute an airflow backend. I'd consider the scheduling of DAGs as a follow on concern to building a Helm chart to wire up the airflow backend. Ideally the airflow backend will be discoverable within k8s such that 'schedule a DAG' is more of k8s one shot job rather than an infrastructural configuration activity. The lifecycle of the back end is different from that of any individual DAG so I'd model them separately.

from kube-airflow.

Absolutely. We wrote a function that will schedule docker containers within kubernetes but that's outside of the scope of this issue.

I was just talking about the type of volume that we should create to ensure the dags are pulled and shared between all of the containers.

from kube-airflow.

I have written logic to apply dag and plugin configmaps to volume mounts on each airflow deployment as a part of the dag/plugin deployment process. The one issue I have run into is that while the volume mounts are updated almost immediately when the configmap changes are applied, the airflow webserver does not refresh the list of DAGs and show the new DAGs that have been added. Is there a good solution to update the webserver?

from kube-airflow.

We were using a sidekick container to pull the dags from a git repo and mount that as the volume. So in our case, we really aren't updating them as often. I guess my thought is you would use an NFS share or something similar if you were going to be doing active dag development. Regarding the webserver... even clicking the refresh icon isn't bringing them up? We've found it sometimes takes the webserver a bit to register the changes.

We did create a configmap for the airflow.cfg file so all of the services could find each other. That config map is also responsible for setting the fernet key. Currently, it creates the secrets for the DB password but that could be changed based on any suggestions.

How did you do configmaps for your dags?

The last part we are working on is with the scheduler restarting. There are a few issues that were opened regarding kubernetes doing the crash loop backoff and we need to modify the entrypoint file to keep the scheduler from restarting so much.

from kube-airflow.

what repo are you doing this development out of?

from kube-airflow.

About the scheduling part, maybe you followed, but there is this promising PR :)

I'll also need an helm chart soonish, and I'll PR the official repo.
If you have any code to share, it would be perfect timing :)

Thanks!

from kube-airflow.

I'm planning to work on this 😃
@pierreozoux Hi, have you by any chance written any code to share?

from kube-airflow.

My priorities have been shuffled :) I think I'll come back to this in January. Sorry :/
But would be happy to review and give feedback!

from kube-airflow.

@pierreozoux Thanks for the reply. No worry - I wish I will come up with something to share with you 😉

from kube-airflow.

May I help on this subject? I'm interesting in integrating Airflow in our Kubernetes cluster, if I may do something to help you, please tell me

from kube-airflow.

@stibbons Hi, thanks for the proposal 👍

I'd greatly appreciate any kind of contribution but probably the most helpful at this stage would be a basic helm chart.

More concretely, even just a translation of https://github.com/mumoshu/kube-airflow/blob/master/airflow.all.yaml into a idiomatic helm chart like seen in https://github.com/kubernetes/charts/tree/master/stable would be greatly appreciated.

from kube-airflow.

Yes, that's my idea as well. First an helm chart for the basic celery airflow, and then a variant with these great patches :)

from kube-airflow.

Hi, everyone. JFYI but see @gsemet's awesome work at helm/charts#3959!

from kube-airflow.

I finally merged the PR @gsemet has kindly submitted.
But he maintains his own fork and a helm chart far better than me!

So, I'll keep this repository for mostly for a reference for a while.
But I believe - the general direction should be treat @gsemet's work as the primary, and contribute any missing parts(if any. probably nothing) to his k8s chart and eventually obsolete this as superseded by it.

from kube-airflow.

Hello. For information we moved toward the official helm repository on this PR: kubernetes/charts/incubator/airflow. For motivated people there are still some issues to fix before official merge on kubernetes/charts !

from kube-airflow.