Comments (13)
sttts
commented on August 12, 2024
/cc @s-urbaniak
from apiserver.
yuchengwu
commented on August 12, 2024
My kubernetes version is 1.9, and I encounter same panic, starting from line 8748 is the first panic log
8748 Trace[6105384]: [75.334µs] [75.334µs] About to List from storage
8749 Trace[6105384]: [5.427484ms] [5.35215ms] Listing from storage done
8750 Trace[6105384]: [14.696485ms] [9.269001ms] Self-linking done
8751 Trace[6105384]: [51.357908193s] [51.343211708s] Writing http response done (1481 items)
8752 Trace[6105384]: [51.357918068s] [9.875µs] END
8753 I0703 18:42:27.727021 1 asm_amd64.s:509] GET /api/v1/endpoints?limit=500&resourceVersion=0: (1m9.740803696s) 200 [[nginx-ingress-controller/v0.0.0 (linux/amd64) kubernetes/$Format] 172.18.28
8753 .14:52610]
8754 E0703 18:42:27.738905 1 runtime.go:66] Observed a panic: &errors.errorString{s:"kill connection/stream"} (kill connection/stream)
8755 /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72
8756 /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
8757 /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
8758 /usr/local/go/src/runtime/asm_amd64.s:509
8759 /usr/local/go/src/runtime/panic.go:491
8760 /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:217
8761 /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:101
8762 /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/waitgroup.go:51
8763 /usr/local/go/src/net/http/server.go:1918
8764 /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/requestinfo.go:45
8765 /usr/local/go/src/net/http/server.go:1918
8766 /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request/requestcontext.go:110
8767 /usr/local/go/src/net/http/server.go:1918
8768 /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/wrap.go:41
8769 /usr/local/go/src/net/http/server.go:1918
8770 /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:199
8771 /usr/local/go/src/net/http/server.go:2619
8772 /usr/local/go/src/net/http/server.go:3164
8773 <autogenerated>:1
8774 /usr/local/go/src/net/http/h2_bundle.go:5462
8775 /usr/local/go/src/net/http/h2_bundle.go:5747
8776 /usr/local/go/src/runtime/asm_amd64.s:2337
8777 E0703 18:42:27.739248 1 wrap.go:34] APIServer panic'd on GET /api/v1/endpoints?limit=500&resourceVersion=0: kill connection/stream
8778 goroutine 1409 [running]:
8779 runtime/debug.Stack(0xa91daa0, 0xc426b0c540, 0x4bc8c84)
8780 /usr/local/go/src/runtime/debug/stack.go:24 +0xa7
8781 k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.WithPanicRecovery.func1.1(0x40c1c60, 0xc4202182d0)
8782 /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/wrap.go:34 +0x74
8783 k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0xc435ca1cf0, 0x1, 0x1)
8784 /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:54 +0xc8
8785 panic(0x40c1c60, 0xc4202182d0)
8786 /usr/local/go/src/runtime/panic.go:491 +0x283
8787 k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.(*baseTimeoutWriter).timeout(0xc42dbb0e00, 0xc4303a6090)
8788 /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:217 +0x190
8789 k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP(0xc429cff800, 0xa91daa0, 0xc426b0c540, 0xc4316a6500)
8790 /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:101 +0x292
8791 k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.WithWaitGroup.func1(0xa91daa0, 0xc426b0c540, 0xc4316a6500)
8792 /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/waitgroup.go:51 +0xeb
8793 net/http.HandlerFunc.ServeHTTP(0xc429884cc0, 0xa91daa0, 0xc426b0c540, 0xc4316a6500)
8794 /usr/local/go/src/net/http/server.go:1918 +0x44
8795 k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithRequestInfo.func1(0xa91daa0, 0xc426b0c540, 0xc4316a6500)
8796 /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/requestinfo.go:45 +0x20f
8797 net/http.HandlerFunc.ServeHTTP(0xc429884d00, 0xa91daa0, 0xc426b0c540, 0xc4316a6500)
8798 /usr/local/go/src/net/http/server.go:1918 +0x44
8799 k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request.WithRequestContext.func1(0xa91daa0, 0xc426b0c540, 0xc4316a6500)
8800 /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request/requestcontext.go:110 +0xcb
8801 net/http.HandlerFunc.ServeHTTP(0xc429cff820, 0xa91daa0, 0xc426b0c540, 0xc4316a6500)
8802 /usr/local/go/src/net/http/server.go:1918 +0x44
8803 k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.WithPanicRecovery.func1(0xa91daa0, 0xc426b0c540, 0xc4316a6500)
8804 /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/wrap.go:41 +0x108
8805 net/http.HandlerFunc.ServeHTTP(0xc429cff840, 0xa91ef20, 0xc43162a040, 0xc4316a6500)
8806 /usr/local/go/src/net/http/server.go:1918 +0x44
8807 k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.(*APIServerHandler).ServeHTTP(0xc422ed2600, 0xa91ef20, 0xc43162a040, 0xc4316a6500)
8808 /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:199 +0x51
8809 net/http.serverHandler.ServeHTTP(0xc42bc2d930, 0xa91ef20, 0xc43162a040, 0xc4316a6500)
8810 /usr/local/go/src/net/http/server.go:2619 +0xb4
8811 net/http.initNPNRequest.ServeHTTP(0xc42b60e380, 0xc42bc2d930, 0xa91ef20, 0xc43162a040, 0xc4316a6500)
8812 /usr/local/go/src/net/http/server.go:3164 +0x9a
8813 net/http.(*initNPNRequest).ServeHTTP(0xc42b471220, 0xa91ef20, 0xc43162a040, 0xc4316a6500)
8814 <autogenerated>:1 +0x63
8815 net/http.(Handler).ServeHTTP-fm(0xa91ef20, 0xc43162a040, 0xc4316a6500)
8816 /usr/local/go/src/net/http/h2_bundle.go:5462 +0x4d
8817 net/http.(*http2serverConn).runHandler(0xc4308a8c40, 0xc43162a040, 0xc4316a6500, 0xc42dbb0d80)
8818 /usr/local/go/src/net/http/h2_bundle.go:5747 +0x89
8819 created by net/http.(*http2serverConn).processHeaders
8820 /usr/local/go/src/net/http/h2_bundle.go:5481 +0x495
8821
8822 I0703 18:42:27.739407 1 logs.go:41] http2: panic serving 172.18.28.14:52610: kill connection/stream
8823 goroutine 1409 [running]:
8824 net/http.(*http2serverConn).runHandler.func1(0xc43162a040, 0xc435ca1faf, 0xc4308a8c40)
8825 /usr/local/go/src/net/http/h2_bundle.go:5740 +0x190
8826 panic(0x40c1c60, 0xc4202182d0)
8827 /usr/local/go/src/runtime/panic.go:491 +0x283
8828 k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0xc435ca1cf0, 0x1, 0x1)
8829 /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x111
8830 panic(0x40c1c60, 0xc4202182d0)
8831 /usr/local/go/src/runtime/panic.go:491 +0x283
8832 k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.(*baseTimeoutWriter).timeout(0xc42dbb0e00, 0xc4303a6090)
8833 /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:217 +0x190
8834 k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP(0xc429cff800, 0xa91daa0, 0xc426b0c540, 0xc4316a6500)
8835 /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:101 +0x292
8836 k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.WithWaitGroup.func1(0xa91daa0, 0xc426b0c540, 0xc4316a6500)
8837 /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/waitgroup.go:51 +0xeb
8838 net/http.HandlerFunc.ServeHTTP(0xc429884cc0, 0xa91daa0, 0xc426b0c540, 0xc4316a6500)
8839 /usr/local/go/src/net/http/server.go:1918 +0x44
8840 k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithRequestInfo.func1(0xa91daa0, 0xc426b0c540, 0xc4316a6500)
8841 /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/requestinfo.go:45 +0x20f
8842 net/http.HandlerFunc.ServeHTTP(0xc429884d00, 0xa91daa0, 0xc426b0c540, 0xc4316a6500)
8843 /usr/local/go/src/net/http/server.go:1918 +0x44
8844 k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request.WithRequestContext.func1(0xa91daa0, 0xc426b0c540, 0xc4316a6500)
8845 /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request/requestcontext.go:110 +0xcb
8846 net/http.HandlerFunc.ServeHTTP(0xc429cff820, 0xa91daa0, 0xc426b0c540, 0xc4316a6500)
8847 /usr/local/go/src/net/http/server.go:1918 +0x44
8848 k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.WithPanicRecovery.func1(0xa91daa0, 0xc426b0c540, 0xc4316a6500)
8849 /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/wrap.go:41 +0x108
8850 net/http.HandlerFunc.ServeHTTP(0xc429cff840, 0xa91ef20, 0xc43162a040, 0xc4316a6500)
8851 /usr/local/go/src/net/http/server.go:1918 +0x44
8852 k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.(*APIServerHandler).ServeHTTP(0xc422ed2600, 0xa91ef20, 0xc43162a040, 0xc4316a6500)
8853 /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:199 +0x51
8854 net/http.serverHandler.ServeHTTP(0xc42bc2d930, 0xa91ef20, 0xc43162a040, 0xc4316a6500)
8855 /usr/local/go/src/net/http/server.go:2619 +0xb4
8856 net/http.initNPNRequest.ServeHTTP(0xc42b60e380, 0xc42bc2d930, 0xa91ef20, 0xc43162a040, 0xc4316a6500)
8857 /usr/local/go/src/net/http/server.go:3164 +0x9a
8858 net/http.(*initNPNRequest).ServeHTTP(0xc42b471220, 0xa91ef20, 0xc43162a040, 0xc4316a6500)
8859 <autogenerated>:1 +0x63
8860 net/http.(Handler).ServeHTTP-fm(0xa91ef20, 0xc43162a040, 0xc4316a6500)
8861 /usr/local/go/src/net/http/h2_bundle.go:5462 +0x4d
8862 net/http.(*http2serverConn).runHandler(0xc4308a8c40, 0xc43162a040, 0xc4316a6500, 0xc42dbb0d80)
8863 /usr/local/go/src/net/http/h2_bundle.go:5747 +0x89
8864 created by net/http.(*http2serverConn).processHeaders
8865 /usr/local/go/src/net/http/h2_bundle.go:5481 +0x495
8866 I0703 18:42:27.746493 1 handler.go:161] kube-aggregator: GET "/apis/extensions/v1beta1/ingresses" satisfied by nonGoRestful
... ...
there are 3 apiserver running simultaneously in our cluster, but only one panics, the panic didn't last too long, after about 4 times, apiserver works normally.
this is the full log apiserverpanic.log , all panics happen before line 10000.
from apiserver.
fejta-bot
commented on August 12, 2024
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale
from apiserver.
fejta-bot
commented on August 12, 2024
Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
from apiserver.
fejta-bot
commented on August 12, 2024
Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close
from apiserver.
k8s-ci-robot
commented on August 12, 2024
@fejta-bot: Closing this issue.
In response to this:
Rotten issues close after 30d of inactivity.
Reopen the issue with/reopen.
Mark the issue as fresh with/remove-lifecycle rotten.Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
from apiserver.
wgliang
commented on August 12, 2024
/reopen
/remove-lifecycle rotten
Facing the same issue.
from apiserver.
k8s-ci-robot
commented on August 12, 2024
@wgliang: Reopened this issue.
In response to this:
/reopen
/remove-lifecycle rottenFacing the same issue.
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
from apiserver.
youngsoo777
commented on August 12, 2024
/reopen
/remove-lifecycle rotten
Facing the same issue...
from apiserver.
fejta-bot
commented on August 12, 2024
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale
from apiserver.
fejta-bot
commented on August 12, 2024
Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
from apiserver.
fejta-bot
commented on August 12, 2024
Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close
from apiserver.
k8s-ci-robot
commented on August 12, 2024
@fejta-bot: Closing this issue.
In response to this:
Rotten issues close after 30d of inactivity.
Reopen the issue with/reopen.
Mark the issue as fresh with/remove-lifecycle rotten.Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
from apiserver.
Related Issues (20)
- apiserver fails json patches with 422 when patch body does not match default json.Marshal encoding HOT 4
- unable to get openapi models: OpenAPIV3 config must not be nil HOT 1
- log error about cert expired - does not show important info HOT 8
- Is it possible to disable TLS in apiserver? I would want istio sidecar to terminate TLS. HOT 4
- Update grpc/otelgrpc dependency to fix CVE-2023-47108 HOT 2
- Segfault when kube-client is not specified.
- Invalid Go version in go.mod HOT 1
- Expose username and group from the appropriate headers HOT 9
- Why exclude resources with errors in the calculation? These errors do not necessarily affect all evaluators HOT 4
- apiserver returns incorrectly results when "Limit" and "LabelSelector" parameters are both set in certain case HOT 7
- Update go-restful module to version >= 3.8.0 due to security issues HOT 5
- OIDC errors HOT 4
- Weird behaviour for remainingItemCount for List HOT 5
- EOL Component HOT 6
- `undefined: unsafe.StringData` on build 1.27-rc.0 with go 1.19 HOT 3
- feature request - audit policy non-resource URL full wildcard support HOT 4
- Query on ContinueToken HOT 8
- Tag v0.28.1 - Issue HOT 6
- Dependency upgrade request - Incompatibility with go.opentelemetry.io/otel version v.1.15.0 and above HOT 5
- New apiserver release that fixes CVE-2023-44487 HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from apiserver.