jmdacruz / celery-k8s-operator Goto Github PK

View Code? Open in Web Editor NEW

3.0 3.0 1.0 116 KB

Kubernetes operator for deploying Python Celery applications

License: MIT License

Dockerfile 3.90% Python 70.96% Shell 25.15%

celery-k8s-operator's People

Contributors

Stargazers

Watchers

Forkers

krzemienski

celery-k8s-operator's Issues

Kubernetes objects vs. YAML files

We started using YAML files on the handler directly, but kopf also supports kubernetes objects generated with the official kubernetes Python SDK. Using objects will tidy up the code a little bit.

operator fails to launch due to permissions

The RBAC configuration is not correct:

[2019-12-11 05:18:28,412] kopf.reactor.running [ERROR   ] Root task 'watcher of celeryapplications.celery.k8s.io' is failed: ClientResponseError(RequestInfo(url=URL('https://10.96.0.1:443/apis/celery.k8s.io/v1beta1/celeryapplications'), method='GET', headers=<CIMultiDictProxy('Host': '10.96.0.1', 'Authorization': 'Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IjdUaU1rWVMtSXd1ZkhzakVFZWpjVFNObHp4VTU1NTAwRmZXWUhlM3V4ZTAifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJjZWxlcnktb3BlcmF0b3IiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlY3JldC5uYW1lIjoiY2VsZXJ5LW9wZXJhdG9yLXRva2VuLTc0OHY5Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImNlbGVyeS1vcGVyYXRvciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImViYmJkZjU5LWY3MjEtNDc0NC1hOTk5LTliZjkyYzc0YWZiNCIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpjZWxlcnktb3BlcmF0b3I6Y2VsZXJ5LW9wZXJhdG9yIn0.C7b9vjQgJ13X2KY6w0n_5ySy7P_xDmNistBmtE0PiZXTCb13kx0cjEzI-eDGxajcOJjnnFUgVODb6xXnhyx0PbaqByiXM_0COCdMps_p4KbKX70AtL2iT22JOsGXVyZE9-QZBJGXb3XD7jKsC8JO2zaOTHquYDU9nQnT8PVNw9lrJTKgct1gMI7Wdi3LCSY3UrUsYzvbs1JOf2tZzb3ABD0JeypeBsnSEPLm6Bq4OYGAz89rzSOzx0IjmMu0RXDYnPgW4d7hsRvBRE68oFuuJWH1sGH0rV3Yg3KkK9f5IcbAQs0JKopE7iOd_65LMoWQ93Gcy3q2ToM3c0nvXEbacA', 'User-Agent': 'kopf/unknown', 'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate')>, real_url=URL('https://10.96.0.1:443/apis/celery.k8s.io/v1beta1/celeryapplications')), (), status=403, message='Forbidden', headers=<CIMultiDictProxy('Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'Date': 'Wed, 11 Dec 2019 05:18:28 GMT', 'Content-Length': '383')>)
Traceback (most recent call last):
  File "/usr/local/bin/kopf", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/kopf/cli.py", line 31, in wrapper
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/decorators.py", line 64, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/kopf/cli.py", line 78, in run
    vault=__controls.vault,
  File "/usr/local/lib/python3.7/site-packages/kopf/reactor/running.py", line 113, in run
    vault=vault,
  File "/usr/local/lib/python3.7/asyncio/base_events.py", line 579, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.7/site-packages/kopf/reactor/running.py", line 155, in operator
    await run_tasks(operator_tasks, ignored=existing_tasks)
  File "/usr/local/lib/python3.7/site-packages/kopf/reactor/running.py", line 341, in run_tasks
    await _reraise(root_done | root_cancelled | hung_done | hung_cancelled)
  File "/usr/local/lib/python3.7/site-packages/kopf/reactor/running.py", line 402, in _reraise
    task.result()  # can raise the regular (non-cancellation) exceptions.
  File "/usr/local/lib/python3.7/site-packages/kopf/reactor/running.py", line 418, in _root_task_checker
    await coro
  File "/usr/local/lib/python3.7/site-packages/kopf/reactor/queueing.py", line 101, in watcher
    async for event in watching.infinite_watch(resource=resource, namespace=namespace):
  File "/usr/local/lib/python3.7/site-packages/kopf/clients/watching.py", line 64, in infinite_watch
    async for event in streaming_watch(resource=resource, namespace=namespace):
  File "/usr/local/lib/python3.7/site-packages/kopf/clients/watching.py", line 80, in streaming_watch
    items, resource_version = await fetching.list_objs_rv(resource=resource, namespace=namespace)
  File "/usr/local/lib/python3.7/site-packages/kopf/clients/auth.py", line 46, in wrapper
    return await fn(*args, **kwargs, session=session)
  File "/usr/local/lib/python3.7/site-packages/kopf/clients/fetching.py", line 101, in list_objs_rv
    response.raise_for_status()
  File "/usr/local/lib/python3.7/site-packages/aiohttp/client_reqrep.py", line 946, in raise_for_status
    headers=self.headers)
aiohttp.client_exceptions.ClientResponseError: 403, message='Forbidden', url=URL('https://10.96.0.1:443/apis/celery.k8s.io/v1beta1/celeryapplications')

Integrate CI/CD using Travis CI

Need a CI/CD service that supports running kind. Apparently Travis CI would support it

State machine for the CeleryApplication status

Need to keep track of the status of child objects (e.g., Deployments), and update the status of the CeleryApplication CRD accordingly (see zalando-incubator/kopf#268 and https://www.youtube.com/watch?v=rN_rQU92T5s&t=875s). Also, we should ideally assess the health of the cluster (at least right after it is created) and update the status of the CeleryApplication. Potential states:

New: Right after the CRD is created, when the operator first reacts to it)
Deployed: When all sub resources have been successfully created. For Deployments, this means that all ReplicaSets are at the desired number of Pods.
Ready: When all celery workers are connected to the messaging broker and available to process tasks.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.