Comments (8)
pacoxu
commented on August 29, 2024
I0225 17:35:11.161300 20814 ooms_tracker.go:284] OOM detected: &Event{ObjectMeta:{gce-scale-cluster-minion-heapster.17b72a4a62c4a7d8 default d60b06ff-11d1-4cb8-be1f-7264e084769b 85768 0 2024-02-25 17:11:22 +0000 UTC map[] map[] [] [] [{node-problem-detector Update v1 2024-02-25 17:11:22 +0000 UTC FieldsV1 {"f:count":{},"f:firstTimestamp":{},"f:involvedObject":{},"f:lastTimestamp":{},"f:message":{},"f:reason":{},"f:source":{"f:component":{},"f:host":{}},"f:type":{}} }]},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:gce-scale-cluster-minion-heapster,UID:gce-scale-cluster-minion-heapster,APIVersion:,ResourceVersion:,FieldPath:,},Reason:OOMKilling,Message:Memory cgroup out of memory: Killed process 4550 (metrics-server) total-vm:834776kB, anon-rss:100100kB, file-rss:33900kB, shmem-rss:0kB, UID:65534 pgtables:400kB oom_score_adj:999,Source:EventSource{Component:kernel-monitor,Host:gce-scale-cluster-minion-heapster,},FirstTimestamp:2024-02-25 17:11:22 +0000 UTC,LastTimestamp:2024-02-25 17:11:22 +0000 UTC,Count:1,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:,ReportingInstance:,}
The original memory limit for NPD pod is 100Mi. It hit the OOM.
It may indict larger memory usage with v0.8.13.
we update it in k/k from 0.8.9 to v0.8.13 in kubernetes/kubernetes#121382 v1.29.
I0225 17:35:11.161300 20814 ooms_tracker.go:284] OOM detected: &Event{ObjectMeta:{gce-scale-cluster-minion-heapster.17b72a4a62c4a7d8 default d60b06ff-11d1-4cb8-be1f-7264e084769b 85768 0 2024-02-25 17:11:22 +0000 UTC <nil> <nil> map[] map[] [] [] [{node-problem-detector Update v1 2024-02-25 17:11:22 +0000 UTC FieldsV1 {"f:count":{},"f:firstTimestamp":{},"f:involvedObject":{},"f:lastTimestamp":{},"f:message":{},"f:reason":{},"f:source":{"f:component":{},"f:host":{}},"f:type":{}} }]},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:gce-scale-cluster-minion-heapster,UID:gce-scale-cluster-minion-heapster,APIVersion:,ResourceVersion:,FieldPath:,},Reason:OOMKilling,Message:Memory cgroup out of memory: Killed process 4550 (metrics-server) total-vm:834776kB, anon-rss:100 100kB, file-rss:33900kB, shmem-rss:0kB, UID:65534 pgtables:400kB oom_score_adj:999,Source:EventSource{Component:kernel-monitor,Host:gce-scale-cluster-minion-heapster,},FirstTimestamp:2024-02-25 17:11:22 +0000 UTC,LastTimestamp:2024-02-25 17:11:22 +0000 UTC,Count:1,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:,ReportingInstance:,}
I0225 17:35:11.161519 20814 ooms_tracker.go:284] OOM detected: &Event{ObjectMeta:{gce-scale-cluster-minion-heapster.17b72a4a642cab59 default 7626405d-a166-40f9-b4fc-7fe01ddb27de 85835 0 2024-02-25 17:11:22 +0000 UTC <nil> <nil> map[] map[] [] [] [{node-problem-detector Update v1 2024-02-25 17:11:22 +0000 UTC FieldsV1 {"f:count":{},"f:firstTimestamp":{},"f:involvedObject":{},"f:lastTimestamp":{},"f:message":{},"f:reason":{},"f:source":{"f:component":{},"f:host":{}},"f:type":{}} }]},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:gce-scale-cluster-minion-heapster,UID:gce-scale-cluster-minion-heapster,APIVersion:,ResourceVersion:,FieldPath:,},Reason:OOMKilling,Message:Memory cgroup out of memory: Killed process 4550 (metrics-server) total-vm:834776kB, anon-rss:101048kB, file-rss:34028kB, shmem-rss:0kB, UID:65534 pgtables:400kB oom_score_adj:999,Source:EventSource{Component:kernel-monitor,Host:gce-scale-cluster-minion-heapster,},FirstTimestamp:2024-02-25 17:11:22 +0000 UTC,LastTimestamp:2024-02-25 17:11:22 +0000 UTC,Count:1,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:,ReportingInstance:,}
I0225 17:35:11.161574 20814 ooms_tracker.go:284] OOM detected: &Event{ObjectMeta:{gce-scale-cluster-minion-heapster.17b72a50237b9584 default 2045b6c3-0c52-4aaa-8cef-bec741102430 155576 0 2024-02-25 17:11:47 +0000 UTC <nil> <nil> map[] map[] [] [] [{node-problem-detector Update v1 2024-02-25 17:11:47 +0000 UTC FieldsV1 {"f:count":{},"f:firstTimestamp":{},"f:involvedObject":{},"f:lastTimestamp":{},"f:message":{},"f:reason":{},"f:source":{"f:component":{},"f:host":{}},"f:type":{}} }]},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:gce-scale-cluster-minion-heapster,UID:gce-scale-cluster-minion-heapster,APIVersion:,ResourceVersion:,FieldPath:,},Reason:OOMKilling,Message:Memory cgroup out of memory: Killed process 5059 (metrics-server) total-vm:829304kB, anon-rss:102212kB, file-rss:32828kB, shmem-rss:0kB, UID:65534 pgtables:380kB oom_score_adj:999,Source:EventSource{Component:kernel-monitor,Host:gce-scale-cluster-minion-heapster,},FirstTimestamp:2024-02-25 17:11:47 +0000 UTC,LastTimestamp:2024-02-25 17:11:47 +0000 UTC,Count:1,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:,ReportingInstance:,}
I0225 17:35:11.161639 20814 ooms_tracker.go:284] OOM detected: &Event{ObjectMeta:{gce-scale-cluster-minion-heapster.17b72a5024e5e2b7 default f318a878-4b13-4e0b-a19d-a58bdbaaa3a8 155636 0 2024-02-25 17:11:47 +0000 UTC <nil> <nil> map[] map[] [] [] [{node-problem-detector Update v1 2024-02-25 17:11:47 +0000 UTC FieldsV1 {"f:count":{},"f:firstTimestamp":{},"f:involvedObject":{},"f:lastTimestamp":{},"f:message":{},"f:reason":{},"f:source":{"f:component":{},"f:host":{}},"f:type":{}} }]},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:gce-scale-cluster-minion-heapster,UID:gce-scale-cluster-minion-heapster,APIVersion:,ResourceVersion:,FieldPath:,},Reason:OOMKilling,Message:Memory cgroup out of memory: Killed process 5072 (metrics-server) total-vm:829304kB, anon-rss:103180kB, file-rss:33136kB, shmem-rss:0kB, UID:65534 pgtables:380kB oom_score_adj:999,Source:EventSource{Component:kernel-monitor,Host:gce-scale-cluster-minion-heapster,},FirstTimestamp:2024-02-25 17:11:47 +0000 UTC,LastTimestamp:2024-02-25 17:11:47 +0000 UTC,Count:1,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:,ReportingInstance:,}
My local npd only use about 10Mi memory.
The problem is not NPD, but metric server. The OOM process is metric server here. I mistakenly read the log.
EDITED: sorry for misreading.
from node-problem-detector.
pacoxu
commented on August 29, 2024
Do we have a benchmark about NPD memory usage?
from node-problem-detector.
pacoxu
commented on August 29, 2024
/close
The log indicated that metric-server is restarted and NPD just detected it.
It is not a NPD issue.
Edited: I may misunderstand the log.
from node-problem-detector.
k8s-ci-robot
commented on August 29, 2024
@pacoxu: Closing this issue.
In response to this:
/close
The log indicated that
metric-serveris restarted and NPD just detected it.It is not a NPD issue.
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
from node-problem-detector.
pacoxu
commented on August 29, 2024
/reopen
The CI still showed the NPD is restarted.
from node-problem-detector.
k8s-ci-robot
commented on August 29, 2024
@pacoxu: Reopened this issue.
In response to this:
/reopen
The CI still showed the NPD is restarted.
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
from node-problem-detector.
wangzhen127
commented on August 29, 2024
/close
Looks like kubernetes/kubernetes#123328 has resolved already. Feel free to reopen if the issue stays.
from node-problem-detector.
k8s-ci-robot
commented on August 29, 2024
@wangzhen127: Closing this issue.
In response to this:
/close
Looks like kubernetes/kubernetes#123328 has resolved already. Feel free to reopen if the issue stays.
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
from node-problem-detector.
Related Issues (20)
- Support revert-pattern in logcounter
- arm64 container contains amd64 binary HOT 1
- What is up with the v0.8.15 tag? HOT 10
- V0.8.15 image is missing log-counter binary HOT 2
- node-problem-detector not able to detect kernel log events for a Kind cluster HOT 2
- Health checker doesn't restart kubelet on EKS HOT 7
- go.mod vulnerabilities fixed in master; when 0.8.16 release HOT 4
- License Scan and Findings HOT 2
- after upgrading npd from v0.8.13 to v0.8.15, containers with unready status: [node-problem-detector] HOT 10
- New CVEs in 0.8.16 HOT 5
- Include automatic CVE scanning in build process HOT 3
- Improve release process HOT 6
- Release v0.8.17 images not done? HOT 3
- Missing test coverage for standalone mode configuration HOT 5
- High CVEs in v0.8.17 HOT 8
- Load rule from configmap HOT 5
- CVE-2022-37434: zlib1g version 1:1.2.11.dfsg-1 HOT 1
- High CVEs in v0.8.18 HOT 8
- draino HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from node-problem-detector.