Comments (6)
ecr-public allows for unauthenticated pulls, which is why that message is only a warning, and I believe may be a red herring.
Looking through the comments I don't think I see anything confirming if this VM has access to the internet, and if that's not enabled that could explain this behavior since we need to pull the container.
Could you please confirm that the networking for QEMU is configured to allow internet access, and that the VM is able to
ping public.ecr.aws
?
Thank you this was really helpful. You were right I was going down the wrong path with the unauthenticated pull warning. ping didn't work, then I realized this was probably a proxy setting issue. It was. Adding the proxy info to my user-data.toml file fixed it.
[settings.network]
https-proxy = "address:port"
no-proxy = ["localhost", "127.0.0.1"]
from bottlerocket.
Taking a look. It is to be expected that you cannot find an ssh service from the console since ssh runs in the admin container.
Can you try checking that the admin container is running with systemctl status [email protected]
?
How confident are you that you have the correct public key, base64 encoded, in your user-data.toml
file and that you have the corresponding private key loaded in your ssh-agent?
Can you show us the net.toml
file?
Thank you.
from bottlerocket.
Taking a look. It is to be expected that you cannot find an ssh service from the console since ssh runs in the admin container.
Can you try checking that the admin container is running with
systemctl status [email protected]
?How confident are you that you have the correct public key, base64 encoded, in your
user-data.toml
file and that you have the corresponding private key loaded in your ssh-agent?Can you show us the
net.toml
file?Thank you.
I'm confident the base64 encoded public key in user-data is correct and I've verified the corresponding private key is loaded in my ssh-agent.
Here's the output from the admin container status. I see "unauthenticated pull". The issue appears to be that I cannot pull "public.ecr.aws/bottlerocket/bottlerocket-admin:v0.11.1" due to failed authorization. Any idea how to troubleshoot this? I have my AWS credentials configured with AWS CLI. I can use AWS CLI commands no problem.
`bash-5.2# systemctl status [email protected]
● [email protected] - Host container: admin
Loaded: loaded (/x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/[email protected]; enabled; preset: enabled)
Active: active (running) since Thu 2024-03-28 18:11:46 UTC; 13s ago
Main PID: 1755 (host-ctr)
Tasks: 11 (limit: 38164)
Memory: 51.2M
CPU: 55ms
CGroup: /system.slice/system-host\x2dcontainers.slice/[email protected]
└─1755 /usr/bin/host-ctr run --container-id=admin --source=public.ecr.aws/bottlerocket/bottlerocket-admin:v0.11.1 --superpowered=true --registry-config=/etc/host-containers/host-ctr.toml
Mar 28 18:11:46 10.0.2.15 systemd[1]: Started Host container: admin.
Mar 28 18:11:46 10.0.2.15 host-ctr[1755]: time="2024-03-28T18:11:46Z" level=info msg="Image does not exist, proceeding to pull image from source." ref="public.ecr.aws/bottlerocket/bottlerocket-admin:v0.11.1"
Mar 28 18:11:59 10.0.2.15 host-ctr[1755]: time="2024-03-28T18:11:59Z" level=warning msg="ecr-public: failed to get authorization token, falling back to default resolver (unauthenticated pull)"`
Here is my net.toml file:
version = 1
[enp0s16]
dhcp4 = true
Thank you.
from bottlerocket.
hmm, what do you see with ctr containers list --address /run/host-containerd/containerd.sock
, are there any containers running? It is weird that systemctl status <>
shows the unit as active.
from bottlerocket.
hmm, what do you see with
ctr containers list --address /run/host-containerd/containerd.sock
, are there any containers running? It is weird thatsystemctl status <>
shows the unit as active.
No, I don't see any containers running.
bash-5.2# ctr containers list
CONTAINER IMAGE RUNTIME
Here's the current output from systemctl status.
`bash-5.2# systemctl status [email protected]
● [email protected] - Host container: admin
Loaded: loaded (/x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/[email protected]; enabled; preset: enabled)
Active: active (running) since Fri 2024-03-29 20:49:14 UTC; 2min 56s ago
Main PID: 1751 (host-ctr)
Tasks: 11 (limit: 38164)
Memory: 52.5M
CPU: 105ms
CGroup: /system.slice/system-host\x2dcontainers.slice/[email protected]
└─1751 /usr/bin/host-ctr run --container-id=admin --source=public.ecr.aws/bottlerocket/bottlerocket-admin:v0.11.1 --superpowered=true --registry-config=/etc/host-containers/host-ctr.toml
Mar 29 20:49:27 10.0.2.15 host-ctr[1751]: time="2024-03-29T20:49:27Z" level=warning msg="ecr-public: failed to get authorization token, falling back to default resolver (unauthenticated pull)"
Mar 29 20:49:57 10.0.2.15 host-ctr[1751]: time="2024-03-29T20:49:57Z" level=info msg="trying next host" error="failed to do request: Head "https://public.ecr.aws/v2/bottlerocket/bottlerocket-admin/manifests/v0.11.1\": dial tcp 99.83.145.10:443: i/o timeout" host=public.ecr.aws
Mar 29 20:49:57 10.0.2.15 host-ctr[1751]: time="2024-03-29T20:49:57Z" level=warning msg="failed to pull image. waiting 4.139s before retrying..." error="failed to resolve reference "public.ecr.aws/bottlerocket/bottlerocket-admin:v0.11.1": failed to do request: Head "https://public.ecr.aws/v2/bottlerocket/bottlerocket-admin/manifests/v0.11.1\": dial tcp 99.83.145.10:443: i/o timeout"
Mar 29 20:50:14 10.0.2.15 host-ctr[1751]: time="2024-03-29T20:50:14Z" level=warning msg="ecr-public: failed to get authorization token, falling back to default resolver (unauthenticated pull)"
Mar 29 20:50:44 10.0.2.15 host-ctr[1751]: time="2024-03-29T20:50:44Z" level=info msg="trying next host" error="failed to do request: Head "https://public.ecr.aws/v2/bottlerocket/bottlerocket-admin/manifests/v0.11.1\": dial tcp 99.83.145.10:443: i/o timeout" host=public.ecr.aws
Mar 29 20:50:44 10.0.2.15 host-ctr[1751]: time="2024-03-29T20:50:44Z" level=warning msg="failed to pull image. waiting 6.393s before retrying..." error="failed to resolve reference "public.ecr.aws/bottlerocket/bottlerocket-admin:v0.11.1": failed to do request: Head "https://public.ecr.aws/v2/bottlerocket/bottlerocket-admin/manifests/v0.11.1\": dial tcp 99.83.145.10:443: i/o timeout"
Mar 29 20:51:02 10.0.2.15 host-ctr[1751]: time="2024-03-29T20:51:02Z" level=warning msg="ecr-public: failed to get authorization token, falling back to default resolver (unauthenticated pull)"
Mar 29 20:51:32 10.0.2.15 host-ctr[1751]: time="2024-03-29T20:51:32Z" level=info msg="trying next host" error="failed to do request: Head "https://public.ecr.aws/v2/bottlerocket/bottlerocket-admin/manifests/v0.11.1\": dial tcp 99.83.145.10:443: i/o timeout" host=public.ecr.aws
Mar 29 20:51:32 10.0.2.15 host-ctr[1751]: time="2024-03-29T20:51:32Z" level=warning msg="failed to pull image. waiting 8.815s before retrying..." error="failed to resolve reference "public.ecr.aws/bottlerocket/bottlerocket-admin:v0.11.1": failed to do request: Head "https://public.ecr.aws/v2/bottlerocket/bottlerocket-admin/manifests/v0.11.1\": dial tcp 99.83.145.10:443: i/o timeout"
Mar 29 20:51:54 10.0.2.15 host-ctr[1751]: time="2024-03-29T20:51:54Z" level=warning msg="ecr-public: failed to get authorization token, falling back to default resolver (unauthenticated pull)"`
from bottlerocket.
ecr-public allows for unauthenticated pulls, which is why that message is only a warning, and I believe may be a red herring.
Looking through the comments I don't think I see anything confirming if this VM has access to the internet, and if that's not enabled that could explain this behavior since we need to pull the container.
Could you please confirm that the networking for QEMU is configured to allow internet access, and that the VM is able to ping public.ecr.aws
?
from bottlerocket.
Related Issues (20)
- settings-committer fails if there are no pending changes HOT 1
- Unable to mount nfs persistent volume from pod running EKS bottlerockt node HOT 3
- v1.20.1 🐫 Tracking Issue HOT 3
- v1.20.2 🤘🏾 Tracking Issue HOT 1
- Publish AMI IDs for K8s with Nvidia support via public SSM parameters, just like other AMIs HOT 4
- core kit migration 🚧 tracking issue HOT 2
- Potential significant max network throughput performance regression HOT 3
- Create an interface for determining the release date of an update HOT 2
- Add the socat package to Bottlerocket
- Need API Setting to allow modify kubelet config for Json logging format HOT 1
- Allow parallel image pulls HOT 5
- `host-ctr` cli crashes when pulling public ECR image HOT 11
- Add `allow2audit` to images HOT 5
- v1.20.3 🐨 Tracking Issue
- Add Kata Containers to images HOT 1
- `cargo make repo` fails after move to `bottlerocket-core-kit` HOT 4
- `dockershim.sock` symlink should be relative HOT 4
- Unresponsive/unreachable Bottlerocket EKS nodes HOT 14
- Enable PodLifecycleSleepAction HOT 1
- Setting cluster-domain has no effect HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from bottlerocket.