Comments (9)
Thank you for this very helpful data. Using this, I was able to reproduce the issue and found the root cause to be a missing field in the HTTPRoute CRD schema. I've added the missing field here #12454 and confirmed that this resolves the issue.
from linkerd2.
from linkerd2.
Another httproute controller-route-user-5186
created and the one above deleted controller-route-user-476722
,
and policy controller keeps throwing hundreds of the same errors:
{"timestamp":"2024-03-21T12:00:12.113301Z","level":"ERROR","fields":{"message":"Failed to patch HTTPRoute","namespace":"my-sandbox","route":"GroupKindName { group: \"policy.linkerd.io\", kind: \"HTTPRoute\", name: \"controller-route-user-476722\" }","error":"ApiError: httproutes.policy.linkerd.io \"controller-route-user-476722\" not found: NotFound (ErrorResponse { status: \"Failure\", message: \"httproutes.policy.linkerd.io \\\"controller-route-user-476722\\\" not found\", reason: \"NotFound\", code: 404 })"},"target":"linkerd_policy_controller_k8s_status::index","spans":[{"name":"status::Controller"}]}
{"timestamp":"2024-03-21T12:00:12.430386Z","level":"ERROR","fields":{"message":"Failed to patch HTTPRoute","namespace":"my-sandbox","route":"GroupKindName { group: \"policy.linkerd.io\", kind: \"HTTPRoute\", name: \"controller-route-user-5186\" }","error":"ApiError: httproutes.policy.linkerd.io \"controller-route-user-5186\" not found: NotFound (ErrorResponse { status: \"Failure\", message: \"httproutes.policy.linkerd.io \\\"controller-route-user-5186\\\" not found\", reason: \"NotFound\", code: 404 })"},"target":"linkerd_policy_controller_k8s_status::index","spans":[{"name":"status::Controller"}]}
and it took about 40 minutes for the status
field to be updated. note the creationTimestamp and status update time.
apiVersion: policy.linkerd.io/v1beta3
kind: HTTPRoute
metadata:
creationTimestamp: '2024-03-21T13:09:58Z'
generation: 1
managedFields:
- apiVersion: policy.linkerd.io/v1beta3
fieldsType: FieldsV1
fieldsV1:
f:spec:
f:parentRefs: {}
f:rules: {}
manager: fabric8
operation: Apply
time: '2024-03-21T13:09:58Z'
- apiVersion: policy.linkerd.io/v1beta3
fieldsType: FieldsV1
fieldsV1:
f:status:
.: {}
f:parents: {}
manager: policy.linkerd.io
operation: Update
subresource: status
time: '2024-03-21T13:49:16Z'
name: controller-route-user-5186
namespace: my-sandbox
resourceVersion: '648189883'
uid: d350c128-3751-4b20-8f85-d0959ffa6c21
selfLink: >-
/apis/policy.linkerd.io/v1beta3/namespaces/my-sandbox/httproutes/controller-route-user-5186
status:
parents:
- conditions:
- lastTransitionTime: '2024-03-21T13:14:16Z'
message: ''
reason: Accepted
status: 'True'
type: Accepted
- lastTransitionTime: '2024-03-21T13:14:16Z'
message: ''
reason: ResolvedRefs
status: 'True'
type: ResolvedRefs
controllerName: linkerd.io/policy-controller
parentRef:
group: core
kind: Service
name: my-controller
namespace: my-sandbox
spec:
parentRefs:
- group: core
kind: Service
name: my-controller
port: 5051
rules:
- backendRefs:
- group: core
kind: Service
name: my-app-0
port: 3004
weight: 1
matches:
- headers:
- name: x-user-id
type: Exact
value: '5186'
path:
type: PathPrefix
value: /
linkerd-destination-56f85576c7-tpx4h_policy.log
from linkerd2.
@aminafshar this looks like it is likely the same issue as #12104 and is fixed in #12215
from linkerd2.
This was fixed in https://github.com/linkerd/linkerd2/releases/tag/edge-24.3.4. Please let us know if issues persist.
from linkerd2.
@adleong , @olix0r
Now we're running edge-24.4.1 (Kubernetes version: v1.28.8),
It seems resource-wise policy controller running normally and memory leak issue resolved
but still we are seeing a long delay of about several minutes between httproute creation and status field update
and we see lots of errors as below
{"timestamp":"2024-04-15T08:18:19.231788Z","level":"ERROR","fields":{"message":"Failed to send HTTPRoute patch","id.namespace":"pangolin","route":"GroupKindName { group: \"policy.linkerd.io\", kind: \"HTTPRoute\", name: \"controller-route-user-4288\" }","error":"no available capacity"},"target":"linkerd_policy_controller_k8s_status::index","spans":[{"name":"status::Index"}]}
![Screenshot 2024-04-15 at 11 17 09](https://private-user-images.githubusercontent.com/3609477/322431004-6ce14182-9fd7-4e8a-b2e6-89a76d5fd4a9.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTkwOTAwMDksIm5iZiI6MTcxOTA4OTcwOSwicGF0aCI6Ii8zNjA5NDc3LzMyMjQzMTAwNC02Y2UxNDE4Mi05ZmQ3LTRlOGEtYjJlNi04OWE3NmQ1ZmQ0YTkucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDYyMiUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA2MjJUMjA1NTA5WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9NGVjN2NiNzM1ODc3NjVjMmE4ZDk3ZGQyNmIzNjMwMzk2MDFiYzc5MzcyMGIwOWI0NTBlODhjM2Q4MWI0ODgyNSZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.YJBAN3u-gUsG2N0sNc1hwb-YT5FIa095KtdGiDSXd3o)
from linkerd2.
As you see memory usage became a flat line for the last 7 hours, and seems the policy controller is just stuck in that state, keeps throwing the same error
2024-04-15T13:53:09+03:00 {"timestamp":"2024-04-15T10:53:09.703054Z","level":"ERROR","fields":{"message":"Failed to send HTTPRoute patch","id.namespace":"pangolin","route":"GroupKindName { group: \"policy.linkerd.io\", kind: \"HTTPRoute\", name: \"controller-route-user-0123\" }","error":"no available capacity"},"target":"linkerd_policy_controller_k8s_status::index","spans":[{"name":"httproutes.policy.linkerd.io"}]}
from linkerd2.
Hi @aminafshar, sorry to hear you're still experiencing this.
Those error messages indicates that the policy controller is generating HTTPRoute status patches more quickly than the kubernetes API can keep up with. The policy controller will only generate a patch for an HTTPRoute if the HTTPRoute's status is out of date and needs to be updated. I've attempted to reproduce this with 1000 HTTPRoutes but I only see patches generated when the HTTTPRoutes are first created and not continuously like you seem to be experiencing. Are HTTPRoutes being created or updated rapidly by some controller or automated process?
If you can provide the output of linkerd diagnostics controller-metrics
, it can help us confirm what we're seeing. If you can also share the yaml formatted output from one of these HTTPRoutes (e.g. kubectl get httproute/X -o yaml
) we can see if anything seems unexpected about the resource itself or its status.
from linkerd2.
Hi @adleong , I asked our developers to provide info on how they create and manage httproutes.
At the time of writing, there are about ~60 httproutes on the cluster and only a few deleted/created recently.
linkerd-destination pods restarted, running for the last ~2hours. Logs and diagnostics output and some recent httproutes yaml output attached.
linkerd-diagnostics-controller-metrics.txt
policy_linkerd-destination-887769595-492pk.log
policy_linkerd-destination-887769595-hdmzp.log
policy_linkerd-destination-887769595-gttn5.log
httproutes.yml.txt
from linkerd2.
Related Issues (20)
- Intermittent routing failures with HTTPRoute HOT 12
- Linkerd-proxy logging full header contents of incoming http requests for log level debug and trace. HOT 3
- Allow port ranges in dynamic authorization policy resources
- Prometheus metrics scrapes of `linkerd-proxy` are not TLS protected (occassionally) HOT 6
- Change default `cr.l5d.io` to `ghcr.io`? HOT 1
- Linkerd Multi-Cluster service-mirroring to give option to mirror EndpointSlices as well HOT 3
- Helm upgrade always changing due to trust root? HOT 2
- Connection refused randomly for pairs of pods HOT 4
- Destination container in the linkerd-destination pod panics when using deployments with headless services
- Connection refused (os error 111) error.sources=[Connection refused (os error 111)] HOT 2
- [FR] - reduce endpoint added/removed logs to debug HOT 4
- `duplicate metrics` in destination controller
- Linkerd is giving 200 or 400 responses for the same un-encoded url request depending on the situation HOT 5
- Linkerd destination repeatedly logging endpoint profile translator errors HOT 3
- Linkerd proxy fails to connect to other proxy HOT 2
- duplicated copies of trust anchor certificate HOT 1
- IPv6 semantics differ from Kubernetes without Linkerd HOT 3
- Helm install documentation refers to incorrect repo HOT 1
- Traefik Router unable to communicate with meshed services when linkerd inbound policy is all-authenticated. HOT 5
- Policy controller fails to watch resources. HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from linkerd2.