Comments (3)
github-actions
commented on August 20, 2024
Hey @usiegl00! Thanks for opening your first issue. We appreciate your contribution and welcome you to our community! We are glad to have you here and to have your input on Contour. You can also join us on our mailing list and in our channel in the Kubernetes Slack Workspace
from contour.
sunjayBhatia
commented on August 20, 2024
I would say this is actually intended behavior, since the leader facility is used to ensure status updates to resources are only ever written by a single contour instance at a time ensuring no conflicts in reported status, all contour instances continue to serve identical xDS configuration to the connected Envoys regardless of their individual leader status
At your own risk, take a look at the existing --disable-leader-election flag: https://projectcontour.io/docs/1.28/configuration/#serve-flags
from contour.
usiegl00
commented on August 20, 2024
Thank you for helping me with this issue.
Setting --disable-leader-election did not prevent contour from crashing when disconnected from the apiserver, and subsequently envoy as well.
Here are the new contour logs:
2024-05-02T01:46:58.231619836Z stderr F time="2024-05-02T01:46:58Z" level=info msg="started HTTP server" address="127.0.0.1:6060" context=debugsvc
2024-05-02T01:46:58.232177747Z stderr F time="2024-05-02T01:46:58Z" level=info msg="started HTTP server" address="0.0.0.0:8000" context=metricsvc
2024-05-02T01:46:58.232211774Z stderr F time="2024-05-02T01:46:58Z" level=info msg="waiting for the initial dag to be built" context=xds
2024-05-02T01:46:58.232225091Z stderr F time="2024-05-02T01:46:58Z" level=info msg="started status update handler" context=StatusUpdateHandler
2024-05-02T01:46:58.232373753Z stderr F time="2024-05-02T01:46:58Z" level=info msg="received a new address for status.loadBalancer" context=loadBalancerStatusWriter loadbalancer-address=192.168.0.1
2024-05-02T01:46:58.333708921Z stderr F time="2024-05-02T01:46:58Z" level=info msg="performing delayed update" context=contourEventHandler last_update=102.148509ms outstanding=13
2024-05-02T01:46:58.434050577Z stderr F time="2024-05-02T01:46:58Z" level=info msg="the initial dag is built" context=xds
2024-05-02T01:46:58.435308924Z stderr F time="2024-05-02T01:46:58Z" level=info msg="started xDS server type: \"contour\"" address="0.0.0.0:8001" context=xds
2024-05-02T01:56:27.112494395Z stderr F time="2024-05-02T01:56:27Z" level=info msg="Stopping and waiting for non leader election runnables" caller="internal.go:516" context=kubernetes
2024-05-02T01:56:27.113164221Z stderr F time="2024-05-02T01:56:27Z" level=info msg="stopped event handler" context=contourEventHandler
2024-05-02T01:56:27.113273139Z stderr F time="2024-05-02T01:56:27Z" level=error msg="terminated HTTP server with error" context=metricsvc error="http: Server closed"
2024-05-02T01:56:27.113391957Z stderr F time="2024-05-02T01:56:27Z" level=error msg="error received after stop sequence was engaged" caller="internal.go:490" context=kubernetes error="http: Server closed"
2024-05-02T01:56:27.113495485Z stderr F time="2024-05-02T01:56:27Z" level=error msg="terminated HTTP server with error" context=debugsvc error="http: Server closed"
2024-05-02T01:56:27.113534012Z stderr F time="2024-05-02T01:56:27Z" level=error msg="error received after stop sequence was engaged" caller="internal.go:490" context=kubernetes error="http: Server closed"
2024-05-02T01:56:27.113785216Z stderr F time="2024-05-02T01:56:27Z" level=info msg="stopped xDS server" address="0.0.0.0:8001" context=xds
2024-05-02T01:56:27.113842337Z stderr F time="2024-05-02T01:56:27Z" level=info msg="Stopping and waiting for leader election runnables" caller="internal.go:520" context=kubernetes
2024-05-02T01:56:27.114009706Z stderr F time="2024-05-02T01:56:27Z" level=info msg="stopped status update handler" context=StatusUpdateHandler
2024-05-02T01:56:27.114057653Z stderr F time="2024-05-02T01:56:27Z" level=info msg="Stopping and waiting for caches" caller="internal.go:526" context=kubernetes
2024-05-02T01:56:27.11482561Z stderr F time="2024-05-02T01:56:27Z" level=info msg="Stopping and waiting for webhooks" caller="internal.go:530" context=kubernetes
2024-05-02T01:56:27.114853454Z stderr F time="2024-05-02T01:56:27Z" level=info msg="Stopping and waiting for HTTP servers" caller="internal.go:533" context=kubernetes
2024-05-02T01:56:27.114863261Z stderr F time="2024-05-02T01:56:27Z" level=info msg="Wait completed, proceeding to shutdown the manager" caller="internal.go:537" context=kubernetes
And these envoy messages appear when it loses connection to contour and stops serving requests:
2024-05-02T02:12:10.529184825Z stderr F [2024-05-02 02:12:10.529][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] StreamRuntime gRPC config stream to contour closed since 942s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: connection timeout
2024-05-02T02:12:10.529209759Z stderr F [2024-05-02 02:12:10.529][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] StreamSecrets gRPC config stream to contour closed since 942s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: connection timeout
2024-05-02T02:12:14.674745043Z stderr F [2024-05-02 02:12:14.674][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] StreamRoutes gRPC config stream to contour closed since 947s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: 111
2024-05-02T02:12:16.766855046Z stderr F [2024-05-02 02:12:16.766][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] StreamClusters gRPC config stream to contour closed since 949s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: 111
2024-05-02T02:12:18.159171337Z stderr F [2024-05-02 02:12:18.158][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] StreamListeners gRPC config stream to contour closed since 950s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: 111
Is there any flag for envoy to keep serving existing traffic upon losing connection to contour?
from contour.
Related Issues (20)
- Cipher configuration issue
- A potential risk in contour that could lead to takeover of the cluster HOT 2
- Bug Alert in Logs HOT 6
- Bump Gateway API support to v1.1.0
- Per IP rate limit? HOT 4
- remove Endpoints translator and feature flag
- Gateway API: implement session persistence + BackendLBPolicy [experimental]
- Gateway API: implement client cert verification [experimental] HOT 1
- Gateway API: implement Gateway params [experimental] HOT 1
- Consider making GRPCRoute a required informer now that it is v1 HOT 1
- Annotate envoy fleet HOT 2
- Contour gateway api provisionner revert annotations HOT 3
- Periodically having `connection failure` HOT 4
- Bug: Remove unnecessary permissions in Helm Charts HOT 2
- Add a separate label for envoy or contour HOT 1
- Add support for HTTP based external auth server HOT 5
- do we have cookie based routing for httpproxy? HOT 4
- Cost saving: Feature request to enable switch to toggle Envoy compression filter HOT 3
- Allow External Auth when Fallback Certificate is used HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from contour.