Comments (8)
cbugneac-nex
commented on September 27, 2024
Just to compliment, this is what I get when checking the readiness endpoint of gatekeeper-audit pod:
curl -v http://100-***-26-129.gatekeeper-system.pod.cluster.local:9090/readyz
* Host 100-***-26-129.gatekeeper-system.pod.cluster.local:9090 was resolved.
* IPv6: (none)
* IPv4: 100.***.26.129
* Trying 100.***.26.129:9090...
* Connected to 100-***-26-129.gatekeeper-system.pod.cluster.local (100.***.26.129) port 9090
> GET /readyz HTTP/1.1
> Host: 100-***-26-129.gatekeeper-system.pod.cluster.local:9090
> User-Agent: curl/8.5.0
> Accept: */*
>
< HTTP/1.1 500 Internal Server Error
< Content-Type: text/plain; charset=utf-8
< X-Content-Type-Options: nosniff
< Date: Fri, 02 Feb 2024 13:19:38 GMT
< Content-Length: 56
<
[-]tracker failed: reason withheld
healthz check failed
Not very informative, which tracks back to this issue #696
from gatekeeper.
commented on September 27, 2024
Have confirmed that the version upgrade will work only if the constraints and templates are never deployed while gatekeeper is installed.
Version bump works if I freshly install gatekeeper v3.12.0, upgrade to v3.13.3 (no installation of templates/constraints).
Version bump will not work if I install gatekeeper v3.12.0, install constraints/templates, uninstall constraints/templates, upgrade to v3.13.3.
Having a look at the logs I can see the following when its failing:
{"level":"error","ts":1706891806.7624226,"msg":"Reconciler error","controller":"constrainttemplate-controller","object":{"name":"k8scontainerlimits"},"namespace":"","name":"k8scontainerlimits","reconcileID":"56bd9d16-5941-49c0-a36d-50c7b1cf6429","error":"Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io \"k8scontainerlimits.constraints.gatekeeper.sh\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:226"}
{"level":"info","ts":1706891806.7961633,"logger":"controller","msg":"handling constraint template status update","process":"constraint_template_status_controller","instance":{"apiVersion":"templates.gatekeeper.sh/v1beta1","kind":"ConstraintTemplate","name":"k8scontainerlimits"}}
from gatekeeper.
commented on September 27, 2024
Another update: I have found that if the following is run before the helm upgrade then it works fine... Seems a bit hacky though
kubectl delete crd -l gatekeeper.sh/system=yes
from gatekeeper.
commented on September 27, 2024
Commenting to keep this alive, any ideas?
from gatekeeper.
maxsmythe
commented on September 27, 2024
If you create a config resource with spec.readiness.statsEnabled = true, what do the logs for the failing audit/webhook pods say?
example config:
apiVersion: config.gatekeeper.sh/v1alpha1
kind: Config
metadata:
name: config
namespace: "gatekeeper-system"
spec:
readiness:
statsEnabled: truefrom gatekeeper.
maxsmythe
commented on September 27, 2024
3.14.0 also contained a readiness fix, maybe upgrading to that version would remediate the issue?
https://github.com/open-policy-agent/gatekeeper/releases/tag/v3.14.0
from gatekeeper.
commented on September 27, 2024
Hi @maxsmythe
Thanks for this info, it actually helped resolve the problem as we had some additional rules applied to the config which were causing the issues. The extensions and v1beta1 ingress were the root cause, removing these allowed us to update as expected.
apiVersion: config.gatekeeper.sh/v1alpha1
kind: Config
metadata:
name: config
namespace: gatekeeper-system
spec:
sync:
syncOnly:
- group: extensions
version: v1beta1
kind: Ingress
- group: networking.k8s.io
version: v1beta1
kind: Ingress
- group: networking.k8s.io
version: v1
kind: Ingress
{{- end }}
from gatekeeper.
maxsmythe
commented on September 27, 2024
Glad I could help!
from gatekeeper.
Related Issues (20)
- Add: app.kubernetes.io/name label to the Deployment object HOT 3
- Migrate psp Templates. HOT 2
- Add a flag for GK validating webhook to defer to vap
- admission webhook "validation.gatekeeper.sh" denied the request HOT 2
- publish images with microarch levels HOT 1
- gatekeeper-controller logs do not display HOT 1
- Metric names mismatch: `*_count` in document, `*_count_total` in actual behavior HOT 1
- OOMKilled as number of constraints grew HOT 4
- doc: Add a page to include all flag information in one place HOT 2
- New example for location value when using complex Labels HOT 1
- 404 Helm chart repo not found HOT 4
- Pass additional info in the mutation request to external data provider HOT 2
- Interpolation in mutation hooks for namespace or other parameters HOT 2
- Upgrade Gatekeeper to use Debian 12 Distroless HOT 3
- WebhookConfigurations(mutating and Validating) causing slow pod creation HOT 1
- Can't use Gator cli to verify opa with external_data HOT 1
- Change chart to only set matchConditions on webhooks when the value parameter is not empty HOT 4
- Support `--log-stats-audit` / `--log-stats-admission` in Helm chart HOT 2
- validation latencies capped at 3 secs even though validatingWebhookTimeoutSeconds set at 5 HOT 2
- gatekeeper max supported qps for a single k8s cluster HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from gatekeeper.