Comments (4)
Hi @dschaaff,
I have had a look at the PSI stuff and here is some of my thoughts towards it.
The PSI accounting is not without overhead. That is the main reason why the kernel provides the ability to disable PSI by default through the mentioned config option. Extra code is run at every sleep/wakeup code paths of the scheduler, which seems scary and hints at default disable is the right way to go. However, As far as I understand the impact of that overhead is of an academic nature and can be seen on micro benchmarks but has yet to be proven to impact real-world workloads.
With all of these trade-off it comes down to some preferring one option over the other. With default disabling we are on the safe side of not introducing overhead to those that may be paranoid and do not use PSI, but are making the lives of those that want to use it a bit harder. The other way around we may make your life easier, but need to have others to disable it explicitly. So which side do we pick?
That being said more recent developments have also included some optimizations for the PSI code that recouped some of the overhead and made these code paths more efficient. With our variants that ship with kernel 6.1 we are in fact not disabling PSI by default. You can use it today without the overhead of the extra reboot by using variants aws-ecs-2*
, *-k8s-1.28*
as well as the *-dev
variants. Please let us know if that is an option for you to unblock your use-case.
In the meantime I will have a look at the overhead on the older kernel versions and if the improvements may have been made available to the 5.10 and 5.15 kernel series. That may inform a decision on changing the default for variants with these kernels.
from bottlerocket.
@dschaaff Thanks for opening this issue! We will research on it and reach back to you. Thanks.
from bottlerocket.
Thanks for the thorough response! We've found the pressure stall metrics valuable for a number of database workloads we run on ec2 directly. I'm glad to hear it is enabled for variants with the 6.1 kernel. We are currently on Kubernetes 1.27 with plans to upgrade to 1.28 beginning of next year. I'm content waiting until then to get the pressure stall info. If it happens to get enabled on the 5.15 kernel versions I'll consider that a treat for me :)
from bottlerocket.
So I have had a look at the improvements that were introduced with the 6.1 kernel through this pull request. Given that the improvements are non-trivial in the 4-9% range and these patches have not been back-ported to any earlier series, I am quite opposed to just enable it by default on the older kernels.
Even though, I have not found reports of the impact of older PSI implementations on real-world workloads, there may be users that are sensitive to these kind of things and that would be in for an unpleasant surprise when just doing a minor update. So we will not change the default for 5.10 and 5.15 based variants to be on the safe side.
from bottlerocket.
Related Issues (20)
- Support for User Namespaces in Kubernetes 1.30 HOT 2
- don't use bootconfig for systemd's unified cgroup hierarchy HOT 1
- v1.19.5 💘 Tracking Issue HOT 1
- pytorch could not detect Nvidia driver on bottlerocket HOT 6
- occasional build failures after extracting subpackages HOT 1
- Looking for aws-dev variant AMI ID HOT 1
- Fail to detect GPU on Bottlerocket v1.19 within AWS g4dn instance HOT 8
- v1.20.0 🐫 Tracking Issue HOT 1
- v1.20.0 update eni-max-pods mapping file HOT 1
- ootb: apiclient needs to be model agnostic HOT 1
- v1.20.0 Host container updates
- Is there any documentation for making bottlerocket work without the internet access to the instances security group ? HOT 1
- kernel-parameters does not accept single-word config options, specifying them causes reboot-loops HOT 3
- BottleRocket NVIDIA EKS Node group wont join EKS Cluster HOT 2
- nvidia-container-cli timeout error when running ECS tasks HOT 1
- Changes to kernel module compression can break certain workflows HOT 15
- Cilium-agent does not start after upgrading to bottlerocket OS 1.20.0 HOT 1
- Host Container Unable to Create Container Task HOT 6
- Collecting logs from EKS Worker Nodes running Bottlerocket AMI when no SSH is enabled HOT 1
- Create symlinks to devices using the device name configured for EBS volumes
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from bottlerocket.