karellen / kubernator Goto Github PK

View Code? Open in Web Editor NEW

4.0 4.0 1.0 253 KB

Kubernator

License: Apache License 2.0

Python 99.87% Dockerfile 0.13%

kubernator's People

Contributors

Stargazers

Watchers

Forkers

jemishp

kubernator's Issues

Introduce improvements to K8SResource collections filtering and querying

K8SResourceList should have something like find and find_jp.
K8SResource should have something like matches and matches_jp.

_jp is "JSON Path"

Move from appdirs to platformdirs

Appdirs not maintained since 2020.

Kubernator terminated with an error: Got an unexpected keyword argument 'field_validation' to method patch_namespaced_custom_object

2023-12-27T01:04:25.412048 kubernator CRITICAL app.py:489 Kubernator terminated with an error: Got an unexpected keyword argument 'field_validation' to method patch_namespaced_custom_object
Traceback (most recent call last):
  File "/home/arcivanov/.local/lib/python3.11/site-packages/kubernator/app.py", line 485, in main
    app.run()
  File "/home/arcivanov/.local/lib/python3.11/site-packages/kubernator/app.py", line 199, in run
    self._run_handlers(KubernatorPlugin.handle_apply, True, context, None)
  File "/home/arcivanov/.local/lib/python3.11/site-packages/kubernator/app.py", line 277, in _run_handlers
    self._set_plugin_context(__reverse, __context, run)
  File "/home/arcivanov/.local/lib/python3.11/site-packages/kubernator/app.py", line 282, in _set_plugin_context
    run(h)
  File "/home/arcivanov/.local/lib/python3.11/site-packages/kubernator/app.py", line 271, in run
    h_f(*args, **kwargs)
  File "/home/arcivanov/.local/lib/python3.11/site-packages/kubernator/plugins/k8s.py", line 280, in handle_apply
    self._apply_resource(dry_run,
  File "/home/arcivanov/.local/lib/python3.11/site-packages/kubernator/plugins/k8s.py", line 424, in _apply_resource
    merged_resource = resource.patch(resource.manifest,
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/arcivanov/.local/lib/python3.11/site-packages/kubernator/plugins/k8s_api.py", line 457, in patch
    resp = rdef.patch(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/arcivanov/.local/lib/python3.11/site-packages/kubernetes/client/api/custom_objects_api.py", line 2941, in patch_namespaced_custom_object
    return self.patch_namespaced_custom_object_with_http_info(group, version, namespace, plural, name, body, **kwargs)  # noqa: E501
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/arcivanov/.local/lib/python3.11/site-packages/kubernetes/client/api/custom_objects_api.py", line 3000, in patch_namespaced_custom_object_with_http_info
    raise ApiTypeError(
kubernetes.client.exceptions.ApiTypeError: Got an unexpected keyword argument 'field_validation' to method patch_namespaced_custom_object

Verify Kubernator works on OSX with Homebrew

Kubernator itself will work, but need to ensure that basic plugins work as well and pull the remote utilities correctly.

BUG: Handle alpha-, beta-, rc- and other K8S client version suffixes

      File "/home/runner/work/kubernator/kubernator/target/dist/kubernator-1.0.16.dev/kubernator/plugins/k8s.py", line 196, in _setup_client
        K8SResource._k8s_client_version = tuple(map(int, pkg_version("kubernetes").split(".")))
                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ValueError: invalid literal for int() with base 10: '0a1'

Minikube vs Kind for Testing

Minikube:

bash-5.2$ time MINIKUBE_HOME=$(readlink -nf .) ./minikube-1.27.1 start --driver=docker --kubernetes-version=v1.20.15
😄  minikube v1.27.1 on Fedora 38
    ▪ MINIKUBE_HOME=/home/arcivanov/Documents/src/karellen/kubernator/test
✨  Using the docker driver based on user configuration
📌  Using Docker driver with root privileges
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=2, Memory=15900MB) ...
🐳  Preparing Kubernetes v1.20.15 on Docker 20.10.18 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

real    0m31.999s
user    0m1.741s
sys     0m0.613s
bash-5.2$ MINIKUBE_HOME=$(readlink -nf .) ./minikube-1.27.1 delete
🔥  Deleting "minikube" in docker ...
🔥  Deleting container "minikube" ...
🔥  Removing /home/arcivanov/Documents/src/karellen/kubernator/test/.minikube/machines/minikube ...
💀  Removed all traces of the "minikube" cluster.
bash-5.2$ time MINIKUBE_HOME=$(readlink -nf .) ./minikube-1.27.1 start --driver=docker --kubernetes-version=v1.20.15 --wait=true
😄  minikube v1.27.1 on Fedora 38
    ▪ MINIKUBE_HOME=/home/arcivanov/Documents/src/karellen/kubernator/test
✨  Using the docker driver based on user configuration
📌  Using Docker driver with root privileges
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=16, Memory=15900MB) ...
🐳  Preparing Kubernetes v1.20.15 on Docker 20.10.18 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

real    2m3.031s
user    0m2.477s
sys     0m0.726s
bash-5.2$ time MINIKUBE_HOME=$(readlink -nf .) ./minikube-1.27.1 start --driver=docker --kubernetes-version=v1.20.15 --wait=api-server --cpus 16
😄  minikube v1.27.1 on Fedora 38
    ▪ MINIKUBE_HOME=/home/arcivanov/Documents/src/karellen/kubernator/test
✨  Using the docker driver based on user configuration
📌  Using Docker driver with root privileges
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=16, Memory=15900MB) ...
🐳  Preparing Kubernetes v1.20.15 on Docker 20.10.18 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

real    0m24.817s
user    0m1.514s
sys     0m0.588s
bash-5.2$ time ./kind create cluster --image kindest/node:v1.20.15 --wait 180s ; kubectl cluster-info --context kind-kind dump
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.20.15) 🖼 
 ✓ Preparing nodes 📦  
 ✓ Writing configuration 📜 
 ✓ Starting control-plane 🕹 
 ✓ Installing CNI 🔌 
 ✓ Installing StorageClass 💾 
 ✓ Waiting ≤ 3m0s for control-plane = Ready ⏳ 
 • Ready after 15s 💚Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Thanks for using kind! 😊
real    0m36.220s
user    0m2.924s
sys     0m2.019s

Tool	Time
Minikube w/ wait=true	123s
Minikube w/ wait=api-server	25s
Minikube w/o wait	32s
Kind w/ wait	36s

Istio plugin refers to a wrong k8s client variable

2023-12-19T15:15:49.729649 kubernator.istio INFO istio.py:187 Saving Istio Operators to /tmp/tmpiq4_vw3z
2023-12-19T15:15:49.754755 kubernator.istio INFO istio.py:191 Running Istio precheck
2023-12-19T15:15:56.196914 kubernator.istio.proc WARNING api.py:628 :heavy_check_mark: No issues found when checking the cluster. Istio is safe to install or upgrade!
2023-12-19T15:15:56.197407 kubernator.istio.proc WARNING api.py:628   To get started, check out https://istio.io/latest/docs/setup/getting-started/
2023-12-19T15:15:56.289242 kubernator.istio.proc WARNING api.py:628 "/tmp/tmpiq4_vw3z" is valid
2023-12-19T15:15:56.291913 kubernator CRITICAL app.py:489 Kubernator terminated with an error: no attribute 'k8s_client'
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/kubernator/app.py", line 485, in main
    app.run()
  File "/usr/local/lib/python3.11/site-packages/kubernator/app.py", line 199, in run
    self._run_handlers(KubernatorPlugin.handle_apply, True, context, None)
  File "/usr/local/lib/python3.11/site-packages/kubernator/app.py", line 277, in _run_handlers
    self._set_plugin_context(__reverse, __context, run)
  File "/usr/local/lib/python3.11/site-packages/kubernator/app.py", line 282, in _set_plugin_context
    run(h)
  File "/usr/local/lib/python3.11/site-packages/kubernator/app.py", line 271, in run
    h_f(*args, **kwargs)
  File "/usr/local/lib/python3.11/site-packages/kubernator/plugins/istio.py", line 197, in handle_apply
    self._operator_init(operators_file, True)
  File "/usr/local/lib/python3.11/site-packages/kubernator/plugins/istio.py", line 210, in _operator_init
    k8s_client = context.k8s.k8s_client
                 ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kubernator/api.py", line 399, in __getattr__
    v = self.__getattr(item)
        ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kubernator/api.py", line 413, in __getattr
    return parent.__getattr(item)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kubernator/api.py", line 414, in __getattr
    raise AttributeError("no attribute %r" % item) from None
AttributeError: no attribute 'k8s_client'

Resources should report their source from the location the template is filled, not where it is defined

2024-01-04T14:47:55.137275 kubernator.k8s ERROR k8s.py:208 FAILED FIELD VALIDATION on resource v1/Service/service-api.general from /home/arcivanov/.../0000-templates/define-templates.tmpl.yaml: annotation service.kubernetes.io/topology-aware-hints is deprecated, please use service.kubernetes.io/topology-mode instead

Add summary of the actions performed in the end

Add a summary of the actions performed in the end. Include information on the number of objects added and modified.

During the dry run, create can raise a 404 in an empty cluster

ServiceAccount sa-name requires Namespace ns-name, but even in dry-run lack of ns-name produces 404.

2023-12-18T22:57:54.987175 kubernator CRITICAL app.py:489 Kubernator terminated with an error: (404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'Audit-Id': '11ee63fd-62c0-461e-b209-d68fd582741f', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '813454a8-a98c-4aaf-944b-1d9499418db7', 'X-Kubernetes-Pf-Prioritylevel-Uid': '227495d7-da81-476e-bb66-75d306e70451', 'Date': 'Tue, 19 Dec 2023 03:57:54 GMT', 'Content-Length': '192'})
HTTP response body: b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"namespaces \\"ns-name\\" not found","reason":"NotFound","details":{"name":"ns-name","kind":"namespaces"},"code":404}\n'
Traceback (most recent call last):
  File "/home/arcivanov/Documents/src/karellen/kubernator/src/main/python/kubernator/plugins/k8s.py", line 357, in _apply_resource
    remote_resource = resource.get()
  File "/home/arcivanov/Documents/src/karellen/kubernator/src/main/python/kubernator/plugins/k8s_api.py", line 398, in get
    return json.loads(self.rdef.get(**kwargs).data)
  File "/home/arcivanov/.pyenv/versions/kubernator/lib/python3.9/site-packages/kubernetes/client/api/core_v1_api.py", line 25269, in read_namespaced_service_account
    return self.read_namespaced_service_account_with_http_info(name, namespace, **kwargs)  # noqa: E501
  File "/home/arcivanov/.pyenv/versions/kubernator/lib/python3.9/site-packages/kubernetes/client/api/core_v1_api.py", line 25356, in read_namespaced_service_account_with_http_info
    return self.api_client.call_api(
  File "/home/arcivanov/.pyenv/versions/kubernator/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 348, in call_api
    return self.__call_api(resource_path, method,
  File "/home/arcivanov/.pyenv/versions/kubernator/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
    response_data = self.request(
  File "/home/arcivanov/.pyenv/versions/kubernator/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 373, in request
    return self.rest_client.GET(url,
  File "/home/arcivanov/.pyenv/versions/kubernator/lib/python3.9/site-packages/kubernetes/client/rest.py", line 244, in GET
    return self.request("GET", url,
  File "/home/arcivanov/.pyenv/versions/kubernator/lib/python3.9/site-packages/kubernetes/client/rest.py", line 238, in request
    raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'Audit-Id': '3db1700d-388e-483f-9acf-dec959fffaa0', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '813454a8-a98c-4aaf-944b-1d9499418db7', 'X-Kubernetes-Pf-Prioritylevel-Uid': '227495d7-da81-476e-bb66-75d306e70451', 'Date': 'Tue, 19 Dec 2023 03:57:54 GMT', 'Content-Length': '218'})
HTTP response body: b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"serviceaccounts \\"sa-name\\" not found","reason":"NotFound","details":{"name":"sa-name","kind":"serviceaccounts"},"code":404}\n'


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/arcivanov/Documents/src/karellen/kubernator/src/main/python/kubernator/app.py", line 485, in main
    app.run()
  File "/home/arcivanov/Documents/src/karellen/kubernator/src/main/python/kubernator/app.py", line 199, in run
    self._run_handlers(KubernatorPlugin.handle_apply, True, context, None)
  File "/home/arcivanov/Documents/src/karellen/kubernator/src/main/python/kubernator/app.py", line 277, in _run_handlers
    self._set_plugin_context(__reverse, __context, run)
  File "/home/arcivanov/Documents/src/karellen/kubernator/src/main/python/kubernator/app.py", line 282, in _set_plugin_context
    run(h)
  File "/home/arcivanov/Documents/src/karellen/kubernator/src/main/python/kubernator/app.py", line 271, in run
    h_f(*args, **kwargs)
  File "/home/arcivanov/Documents/src/karellen/kubernator/src/main/python/kubernator/plugins/k8s.py", line 253, in handle_apply
    self._apply_resource(dry_run,
  File "/home/arcivanov/Documents/src/karellen/kubernator/src/main/python/kubernator/plugins/k8s.py", line 361, in _apply_resource
    create()
  File "/home/arcivanov/Documents/src/karellen/kubernator/src/main/python/kubernator/plugins/k8s.py", line 345, in create
    create_func()
  File "/home/arcivanov/Documents/src/karellen/kubernator/src/main/python/kubernator/plugins/k8s_api.py", line 410, in create
    return json.loads(rdef.create(**kwargs).data)
  File "/home/arcivanov/.pyenv/versions/kubernator/lib/python3.9/site-packages/kubernetes/client/api/core_v1_api.py", line 8518, in create_namespaced_service_account
    return self.create_namespaced_service_account_with_http_info(namespace, body, **kwargs)  # noqa: E501
  File "/home/arcivanov/.pyenv/versions/kubernator/lib/python3.9/site-packages/kubernetes/client/api/core_v1_api.py", line 8617, in create_namespaced_service_account_with_http_info
    return self.api_client.call_api(
  File "/home/arcivanov/.pyenv/versions/kubernator/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 348, in call_api
    return self.__call_api(resource_path, method,
  File "/home/arcivanov/.pyenv/versions/kubernator/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
    response_data = self.request(
  File "/home/arcivanov/.pyenv/versions/kubernator/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 391, in request
    return self.rest_client.POST(url,
  File "/home/arcivanov/.pyenv/versions/kubernator/lib/python3.9/site-packages/kubernetes/client/rest.py", line 279, in POST
    return self.request("POST", url,
  File "/home/arcivanov/.pyenv/versions/kubernator/lib/python3.9/site-packages/kubernetes/client/rest.py", line 238, in request
    raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'Audit-Id': '11ee63fd-62c0-461e-b209-d68fd582741f', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '813454a8-a98c-4aaf-944b-1d9499418db7', 'X-Kubernetes-Pf-Prioritylevel-Uid': '227495d7-da81-476e-bb66-75d306e70451', 'Date': 'Tue, 19 Dec 2023 03:57:54 GMT', 'Content-Length': '192'})
HTTP response body: b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"namespaces \\"ns-name\\" not found","reason":"NotFound","details":{"name":"ns-name","kind":"namespaces"},"code":404}\n'

MacOS Github Runners don't have enough space for minikube

Consider moving to MacOS self-hosted runner image

PATCH protocol violation

2023-12-18T19:03:30.372439 kubernator CRITICAL app.py:453 Kubernator terminated with an error: (400)
Reason: Bad Request
HTTP response headers: HTTPHeaderDict({'Audit-Id': '24a2c5f9-4a41-4d50-afe4-d9b062763f9f', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '53190751-7653-410e-bfc2-1b4d719bb176', 'X-Kubernetes-Pf-Prioritylevel-Uid': '6471b8cf-26e3-407f-9af6-cbea34b51740', 'Date': 'Mon, 18 Dec 2023 19:03:30 GMT', 'Content-Length': '257'})
HTTP response body: b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"error decoding YAML: error unmarshaling JSON: while decoding JSON: json: cannot unmarshal string into Go value of type map[string]interface {}","reason":"BadRequest","code":400}\n'

Switching branch of a cached git repository results in a failure

Switching branches in the cache path results in the following:

2023-12-20T12:18:11.390570 kubernator.proc TRACE proc.py:72 Starting ['git', 'status']
2023-12-20T12:18:11.401337 kubernator.proc TRACE proc.py:72 Starting ['git', 'checkout', 'dev']
2023-12-20T12:18:11.408043 kubernator.repository.git INFO api.py:628 error: pathspec 'dev' did not match any file(s) known to git

Enable testing on K8S 1.30.x, Istio 1.22

Force strict field validation and consider dry-run=All

https://kubernetes.io/docs/reference/using-api/api-concepts/#setting-the-field-validation-level

The field validation level is set by the fieldValidation query parameter.

Default to strict.

Allow overwritting with a warning and interpret warnings.

Improve Handling of the 400 error and beautify response logging processing

HTTP response body: b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"admission webhook \\"[validation.istio.io](http://validation.istio.io/)\\" denied the request: configuration is invalid: Envoy filter: can\'t unmarshal Any nested proto [type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager](http://type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager): json: cannot unmarshal object into Go value of type []json.RawMessage","code":400}\n'

Git fetch refuses to clobber tags in the git cache

2023-12-29T22:36:14.093888 kubernator.repository.git INFO api.py:629  ! [rejected]        release-1.0-1.3.10.5 -> release-1.0-1.3.10.5  (would clobber existing tag)
2023-12-29T22:36:14.131684 kubernator CRITICAL app.py:489 Kubernator terminated with an error: Command '['git', 'fetch', '-pPt']' returned non-zero exit status 1.

Failure to clear/remove annotations and other collections with server-side patch

This is caused by kubernetes/kubernetes#122648

yaml.constructor.ConstructorError: could not determine a constructor for the tag 'tag:yaml.org,2002:value' in "<unicode string>", line 5675, column 29:

Caused by yaml/pyyaml/#683 and related to prometheus-operator/prometheus-operator#6318

Adopting a workaround from: yaml/pyyaml#683 (comment)

Istio: Missing IstioOperator CRD in 1.21.0+

requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://raw.githubusercontent.com/istio/istio/1.21.0/manifests/charts/base/crds/crd-operator.yaml

Reinstate `field_validation` for `CustomObjectApi` objects once K8S Client is fixed

See kubernetes-client/gen#259

Plugin discovery and activation?

Currently the plugins are enumerated and activated statically.
There is a need to discover plugins dynamically.
Dynamic discovery of plugins raises the following concerns:

How to discover plugins: entry points or subclasses or both?
In what order to initialize the plugins that may be interdependent? Manual order, dependency tree etc?
Which of the discovered plugins to initialize? All discovered or manually specified? If manually specified where are they specified? Is a new Ktor initialization phase and/or file required?