cloudfoundry / grootfs Goto Github PK
View Code? Open in Web Editor NEWGarden root file system
License: Apache License 2.0
Garden root file system
License: Apache License 2.0
Hey,
I think I'm hitting a bug, or at least some interesting behaviour with grootfs, when trying to use Cloudfoundry. I see this error when trying to start an app that uses a Docker image.
2018-07-01T17:37:16.13+0100 [CELL/0] ERR Cell 8ebbfec3-2a77-4730-9690-59563f35d4bb failed to create container for instance ad2432b5-1b19-485f-7a1c-90fe: running image plugin create: pulling the image: unpacking layer `sha256:932d501be197d54ce2f98016355f9dddbabc548b8cad866711d5c850e4811749`: creating directory `gnu/store/1r3dlhi2vasb8yw630728jlrk40mygj1-bash-static-4.4.19`: mkdir gnu/store/1r3dlhi2vasb8yw630728jlrk40mygj1-bash-static-4.4.19: no such file or directory
2018-07-01T17:37:16.13+0100 [CELL/0] ERR : exit status 1
If you peek in to the layer, I think I see what is causing the issue:
→ tar --list --file=layer.tar | head
gnu/store/1r3dlhi2vasb8yw630728jlrk40mygj1-bash-static-4.4.19/
gnu/store/1r3dlhi2vasb8yw630728jlrk40mygj1-bash-static-4.4.19/bin/
gnu/store/1r3dlhi2vasb8yw630728jlrk40mygj1-bash-static-4.4.19/bin/bash
gnu/store/1r3dlhi2vasb8yw630728jlrk40mygj1-bash-static-4.4.19/bin/sh
...
I'm no expert on the tar format, but I'll include a simple example of generating a tar archive to demonstrate this is a possibility in a comment.
Maybe switching from using os.Mkdir
[1] to os.MkdirAll
would guard against this possibility?
1: https://github.com/cloudfoundry/grootfs/blob/master/base_image_puller/unpacker/tar.go#L279
Thanks,
Chris
Hi,
we are on cf-deployment v1.9.0, garden-runc v1.11.0. We see in our environment with 450 cells about 12k metrics per seconds only related to origin:"grootfs"
. For now this is not causing a problem for us, since we are just filtering out corresponding metrics.
The top valueMetrics
in our environment within about 30s:
5093 origin:"grootfs"valueMetric:<name:"memoryStats.lastGCPauseTimeNS"
5142 origin:"grootfs"valueMetric:<name:"memoryStats.numBytesAllocatedStack"
5150 origin:"grootfs"valueMetric:<name:"memoryStats.numBytesAllocated"
5160 origin:"grootfs"valueMetric:<name:"numCPUS"
5170 origin:"grootfs"valueMetric:<name:"memoryStats.numFrees"
5170 origin:"grootfs"valueMetric:<name:"memoryStats.numMallocs"
5187 origin:"grootfs"valueMetric:<name:"memoryStats.numBytesAllocatedHeap"
5203 origin:"grootfs"valueMetric:<name:"ImageStatsTime"
5223 origin:"grootfs"valueMetric:<name:"numGoRoutines"
These metrics are coming per cell, in cases even multiple times per second.
The top counterEvents
in our enviroment within about 30s:
...[others not related to `grootfs`]
4523 origin:"grootfs"counterEvent:<name:"grootfs-stats.run"
4557 origin:"grootfs"counterEvent:<name:"grootfs-stats.run.success"
Wouldn't the "grootfs-stats.run"
entries be more suitable for a debug log message instead of a metric?
Hi,
I am facing the following error while running make cf command on SLES12 os with s390x architecture.
compilation-grootfs > store/filesystems/filesystems.go:30:17: invalid operation: statfs.Type != fsType (mismatched types uint32 and int64)
compilation-grootfs > make: *** [Makefile:8: all] Error 2
Dependabot can't resolve your Go dependency files.
As a result, Dependabot couldn't update your dependencies.
The error Dependabot encountered was:
code.cloudfoundry.org/grootfs/commands: cannot find module providing package code.cloudfoundry.org/grootfs/commands
code.cloudfoundry.org/grootfs/commands/config: cannot find module providing package code.cloudfoundry.org/grootfs/commands/config
code.cloudfoundry.org/grootfs/store: cannot find module providing package code.cloudfoundry.org/grootfs/store
If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.
Recently during diego cell update, we noticed an issue where GrootFS was trying to download Docker Image Layers, but this took 30 and 70 minutes (for two instances of the app). As a result the call failed and this exception was written in the logs
failed to create container: running image plugin create: pulling the image: streaming blob `sha256:7d92222c2b4c8e11483aa2b207d03f61c86433db5076c29e53e890e9e2152ddc`: writing blob to tempfile: read tcp 10.0.201.16:51246->104.18.124.25:443: read: connection reset by peer : exit status 1, reason: CRASHED
The same application instances later started and downloaded their blobs within few seconds. All in all docker blob downloads take up to 60s on our busiest landscapes, where up to 7000 LRPs (some of them docker) are updated at the same time.
For the customer this was recoded as downtime, as the new instances of the app did not manage to start on time, and the old one was killed after the rep evacuation timeout.
Tracing the issue in the code led to this part
grootfs/fetcher/layer_fetcher/source/layer_source.go
Lines 209 to 210 in 5d15287
for i := 0; i < MAX_DOCKER_RETRIES; i++ {
logger.Debug(fmt.Sprintf("attempt-get-blob-%d", i+1))
blob, size, e := imgSrc.GetBlob(context.TODO(), blobInfo, none.NoCache)
if e == nil {
logger.Debug("attempt-get-blob-success")
return blob, size, nil
}
err = e
logger.Error("attempt-get-blob-failed", err)
}
func (s *dockerImageSource) GetBlob(ctx context.Context, info types.BlobInfo, cache types.BlobInfoCache) (io.ReadCloser, int64, error) {
if len(info.URLs) != 0 {
return s.getExternalBlob(ctx, info.URLs)
}
path := fmt.Sprintf(blobsPath, reference.Path(s.ref.ref), info.Digest.String())
logrus.Debugf("Downloading %s", path)
res, err := s.c.makeRequest(ctx, "GET", path, nil, nil, v2Auth, nil)
if err != nil {
return nil, 0, err
}
So it looks like it is time to take a look on the context.TODO()
part and add some configurable timeout
E.g. default is no timeout, and via some config property, one could set it to some value - e.g. 60s.
This way the retry logic would handle such requests and quickly download the blob
If you agree, we will be happy to provide a PR
Regards, Vladimir
Dependabot can't resolve your Go dependency files.
As a result, Dependabot couldn't update your dependencies.
The error Dependabot encountered was:
code.cloudfoundry.org/grootfs/commands: cannot find module providing package code.cloudfoundry.org/grootfs/commands
code.cloudfoundry.org/grootfs imports
code.cloudfoundry.org/grootfs/commands/config: cannot find module providing package code.cloudfoundry.org/grootfs/commands/config
code.cloudfoundry.org/grootfs imports
code.cloudfoundry.org/grootfs/store: cannot find module providing package code.cloudfoundry.org/grootfs/store
go: downloading code.cloudfoundry.org/lager v1.1.0
If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.
Since bumping golang to 1.17, grootfs has been regularly failing:
Fix it.
transferred to Garden-Wiki
Actual behavior
I am currently migrating my Dockerfiles to be build with Kaniko (V16 and V19 have been tried). The following error occurs in several of my Dockerfiles when starting in Cloud Foundry with cf push
(Dockerimage build with kaniko: thebohne/kaniko:kaniko
):
15:22:15.270: [CELL.0] Cell 7412a641-2c74-4f99-9cc0-788f7c4536fd failed to create container for instance 640101cd-af6c-467a-4b4e-d704: running image plugin create: pulling the image: unpacking layer `sha256:33db33d935721dfdf882423ee0e4d68410850032bc4e45779f7d746816710787`: failed to unpack: : waiting for the reexec command failed: unpacking-failed: opening target whiteout directory: open test: no such file or directory
15:22:15.270: [CELL.0] : exit status 1
or in another image:
Cell b87239a8-1b05-4ace-93b3-8a29484886d5 failed to create container for instance d6e31d31-7947-4ca7-7eda-651b: OCI runtime create failed: container_linux.go:346: starting container process caused "process_linux.go:449: container init caused \"rootfs_linux.go:58: mounting \\\"/var/vcap/data/garden/bin/init\\\" to rootfs \\\"/var/vcap/data/grootfs/store/unprivileged/images/d6e31d31-7947-4ca7-7eda-651b/rootfs\\\" at \\\"/var/vcap/data/grootfs/store/unprivileged/images/d6e31d31-7947-4ca7-7eda-651b/rootfs/tmp/garden-init\\\" caused \\\"open /var/vcap/data/grootfs/store/unprivileged/images/d6e31d31-7947-4ca7-7eda-651b/rootfs/tmp/garden-init: permission denied\\\"\"": unknown
Expected behavior
The expected behavior looks like this when building with Docker (docker build .
) and starting with cf push
(Dockerimage build with docker: thebohne/kaniko:docker
):
15:49:29.176: [CELL.0] Cell f3551e37-379f-454a-81d3-389a4c6d12a1 successfully created container for instance 77902562-a6aa-4aaf-7659-d8b3
15:49:31.431: [APP/PROC/WEB.0] Exit status 143
To Reproduce
For you, I wrote a minimal Dockerfile:
FROM debian:stretch
RUN mkdir /test
RUN echo "TEST" > /test/testfile
RUN mv /test/testfile /usr/local/bin
The same error should occur with other base images as well.
I am running the following commands in my Jenkinspipeline (there should be no difference when running on your local maschine):
/kaniko/executor --destination=thebohne/kaniko:kaniko --context=${WORKSPACE} --dockerfile=src/main/docker/Dockerfile
or with --cache=true
CF_DOCKER_PASSWORD=${DOCKER_PASSWORD} cf push ${NAME} --docker-image thebohne/kaniko:kaniko --docker-username ${USERNAME} -u process
Test Matrix
Here a little matrix of what I tested:
Run with Docker | Run in Cloud Foundry | |
---|---|---|
Build with Docker | ✔️ | ✔️ |
Build with Kaniko | ✔️ | ❌ |
Else
So I don't know where to post this issue (I hope I'm in the right repo 😁) because there's a link between Kaniko and Cloud Foundry.
Here is the link to the same issue that was published in the Kaniko repo: GoogleContainerTools/kaniko#1149
Thanks for you help :) if you need additional information please write a comment :)
I'm trying to deploy bosh-lite. downloadLayer
function tries to decompress the downloaded tar archive
however bosh-warden-cpi-release
expects to see the archive and I get an error below:
Creating VM with agent ID '{{eb2931fc-1e68-44ef-8492-876b7f01647f}}': Creating container: running image plugin create: invalid base image: directory provided instead of a tar file
I tried to work this around by packing the directory back to the tar:
mv /var/vcap/store/warden_cpi/stemcells/06a29be3-73c4-4d61-5871-5a49a32f313d /var/vcap/store/warden_cpi/stemcells/06a29be3-73c4-4d61-5871-5a49a32f313d.dir
cd /var/vcap/store/warden_cpi/stemcells/06a29be3-73c4-4d61-5871-5a49a32f313d.dir
tar -cf ../06a29be3-73c4-4d61-5871-5a49a32f313d .
Did I miss a config option for warden or grootfs?
@claurence : Pivotal uses GITBOT to synchronize Github issues and pull requests with Pivotal Tracker.
Please add your new repo to the GITBOT config-prouction.yml
in the Gitbot configuration repo
If there are any questions, please reach out to [email protected]
Docker supports a feature called "Content trust" : https://docs.docker.com/engine/security/trust/content_trust/ where the docker images are verified against a signature when they are downloaded from the registry.
Some tenants are requesting us this feature, asking to enable this check per application.
Grootfs can validate the image as it gets downloaded. The challenge is how the user can enable this feature and how the rest of the cloudfoundry stack should handle the situation.
Maybe a custom parameter in the docker url? e.g. docker://myregistry/image?content_trust=true
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.