cloudfoundry / grootfs Goto Github PK

View Code? Open in Web Editor NEW

42.0 29.0 19.0 78.54 MB

Garden root file system

License: Apache License 2.0

Shell 0.89% Makefile 0.34% Go 98.52% Dockerfile 0.25%

garden cloud-foundry rootfs cff-wg-app-runtime-platform

grootfs's People

Contributors

Stargazers

Watchers

Forkers

digideskio masters-of-cats akihirosuda jmcarp windofthesky rhardt-pivotal suse rohitsakala licshire lebentle 7hunderbird stackriot-labs isabella232 klapkov michaelyu0906 sap-contributions ebroberson geofffranks

grootfs's Issues

Issue with unpacking some tar archives

Hey,

I think I'm hitting a bug, or at least some interesting behaviour with grootfs, when trying to use Cloudfoundry. I see this error when trying to start an app that uses a Docker image.

2018-07-01T17:37:16.13+0100 [CELL/0] ERR Cell 8ebbfec3-2a77-4730-9690-59563f35d4bb failed to create container for instance ad2432b5-1b19-485f-7a1c-90fe: running image plugin create: pulling the image: unpacking layer `sha256:932d501be197d54ce2f98016355f9dddbabc548b8cad866711d5c850e4811749`: creating directory `gnu/store/1r3dlhi2vasb8yw630728jlrk40mygj1-bash-static-4.4.19`: mkdir gnu/store/1r3dlhi2vasb8yw630728jlrk40mygj1-bash-static-4.4.19: no such file or directory
2018-07-01T17:37:16.13+0100 [CELL/0] ERR : exit status 1

If you peek in to the layer, I think I see what is causing the issue:

→ tar --list --file=layer.tar | head
gnu/store/1r3dlhi2vasb8yw630728jlrk40mygj1-bash-static-4.4.19/
gnu/store/1r3dlhi2vasb8yw630728jlrk40mygj1-bash-static-4.4.19/bin/
gnu/store/1r3dlhi2vasb8yw630728jlrk40mygj1-bash-static-4.4.19/bin/bash
gnu/store/1r3dlhi2vasb8yw630728jlrk40mygj1-bash-static-4.4.19/bin/sh
...

I'm no expert on the tar format, but I'll include a simple example of generating a tar archive to demonstrate this is a possibility in a comment.

Maybe switching from using os.Mkdir [1] to os.MkdirAll would guard against this possibility?

1: https://github.com/cloudfoundry/grootfs/blob/master/base_image_puller/unpacker/tar.go#L279

Thanks,

Chris

High amount of metrics

Hi,

we are on cf-deployment v1.9.0, garden-runc v1.11.0. We see in our environment with 450 cells about 12k metrics per seconds only related to origin:"grootfs". For now this is not causing a problem for us, since we are just filtering out corresponding metrics.

The top valueMetrics in our environment within about 30s:

   5093 origin:"grootfs"valueMetric:<name:"memoryStats.lastGCPauseTimeNS"
   5142 origin:"grootfs"valueMetric:<name:"memoryStats.numBytesAllocatedStack"
   5150 origin:"grootfs"valueMetric:<name:"memoryStats.numBytesAllocated"
   5160 origin:"grootfs"valueMetric:<name:"numCPUS"
   5170 origin:"grootfs"valueMetric:<name:"memoryStats.numFrees"
   5170 origin:"grootfs"valueMetric:<name:"memoryStats.numMallocs"
   5187 origin:"grootfs"valueMetric:<name:"memoryStats.numBytesAllocatedHeap"
   5203 origin:"grootfs"valueMetric:<name:"ImageStatsTime"
   5223 origin:"grootfs"valueMetric:<name:"numGoRoutines"

These metrics are coming per cell, in cases even multiple times per second.

The top counterEvents in our enviroment within about 30s:

    ...[others not related to `grootfs`]
   4523 origin:"grootfs"counterEvent:<name:"grootfs-stats.run"
   4557 origin:"grootfs"counterEvent:<name:"grootfs-stats.run.success"

Wouldn't the "grootfs-stats.run" entries be more suitable for a debug log message instead of a metric?

Error while running make cf command.

Hi,

I am facing the following error while running make cf command on SLES12 os with s390x architecture.

compilation-grootfs > store/filesystems/filesystems.go:30:17: invalid operation: statfs.Type != fsType (mismatched types uint32 and int64)
compilation-grootfs > make: *** [Makefile:8: all] Error 2

Dependabot can't resolve your Go dependency files

Dependabot can't resolve your Go dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

code.cloudfoundry.org/grootfs/commands: cannot find module providing package code.cloudfoundry.org/grootfs/commands
	code.cloudfoundry.org/grootfs/commands/config: cannot find module providing package code.cloudfoundry.org/grootfs/commands/config
	code.cloudfoundry.org/grootfs/store: cannot find module providing package code.cloudfoundry.org/grootfs/store

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

Bump github.com/docker/docker from 20.10.7+incompatible to 20.10.8+incompatible

No timeout when downloading Docker image Layers

Recently during diego cell update, we noticed an issue where GrootFS was trying to download Docker Image Layers, but this took 30 and 70 minutes (for two instances of the app). As a result the call failed and this exception was written in the logs

 failed to create container: running image plugin create: pulling the image: streaming blob `sha256:7d92222c2b4c8e11483aa2b207d03f61c86433db5076c29e53e890e9e2152ddc`: writing blob to tempfile: read tcp 10.0.201.16:51246->104.18.124.25:443: read: connection reset by peer : exit status 1, reason: CRASHED

The same application instances later started and downloaded their blobs within few seconds. All in all docker blob downloads take up to 60s on our busiest landscapes, where up to 7000 LRPs (some of them docker) are updated at the same time.

For the customer this was recoded as downtime, as the new instances of the app did not manage to start on time, and the old one was killed after the rep evacuation timeout.

Tracing the issue in the code led to this part

grootfs/fetcher/layer_fetcher/source/layer_source.go

Lines 209 to 210 in 5d15287

 logger.Debug(fmt.Sprintf("attempt-get-blob-%d", i+1)) 

 blob, size, e := imgSrc.GetBlob(context.TODO(), blobInfo, none.NoCache)

	for i := 0; i < MAX_DOCKER_RETRIES; i++ {
		logger.Debug(fmt.Sprintf("attempt-get-blob-%d", i+1))
		blob, size, e := imgSrc.GetBlob(context.TODO(), blobInfo, none.NoCache)
		if e == nil {
			logger.Debug("attempt-get-blob-success")
			return blob, size, nil
		}
		err = e
		logger.Error("attempt-get-blob-failed", err)
	}

And Get Blob (https://github.com/cloudfoundry/diego-release/blob/c256419f9d0878c3bff515bdb0a80eb49281e8b9/src/code.cloudfoundry.org/vendor/github.com/containers/image/docker/docker_image_src.go#L224-L234)

func (s *dockerImageSource) GetBlob(ctx context.Context, info types.BlobInfo, cache types.BlobInfoCache) (io.ReadCloser, int64, error) {
	if len(info.URLs) != 0 {
		return s.getExternalBlob(ctx, info.URLs)
	}

	path := fmt.Sprintf(blobsPath, reference.Path(s.ref.ref), info.Digest.String())
	logrus.Debugf("Downloading %s", path)
	res, err := s.c.makeRequest(ctx, "GET", path, nil, nil, v2Auth, nil)
	if err != nil {
		return nil, 0, err
	}

So it looks like it is time to take a look on the context.TODO() part and add some configurable timeout
E.g. default is no timeout, and via some config property, one could set it to some value - e.g. 60s.
This way the retry logic would handle such requests and quickly download the blob

If you agree, we will be happy to provide a PR

Regards, Vladimir

Dependabot can't resolve your Go dependency files

Dependabot can't resolve your Go dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

code.cloudfoundry.org/grootfs/commands: cannot find module providing package code.cloudfoundry.org/grootfs/commands
code.cloudfoundry.org/grootfs imports
	code.cloudfoundry.org/grootfs/commands/config: cannot find module providing package code.cloudfoundry.org/grootfs/commands/config
code.cloudfoundry.org/grootfs imports
	code.cloudfoundry.org/grootfs/store: cannot find module providing package code.cloudfoundry.org/grootfs/store
go: downloading code.cloudfoundry.org/lager v1.1.0

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

grootfs flakes

Since bumping golang to 1.17, grootfs has been regularly failing:

Fix it.

Onboard Georgi

transferred to Garden-Wiki

failed to unpack: : waiting for the reexec command failed

Actual behavior
I am currently migrating my Dockerfiles to be build with Kaniko (V16 and V19 have been tried). The following error occurs in several of my Dockerfiles when starting in Cloud Foundry with cf push (Dockerimage build with kaniko: thebohne/kaniko:kaniko):

15:22:15.270: [CELL.0] Cell 7412a641-2c74-4f99-9cc0-788f7c4536fd failed to create container for instance 640101cd-af6c-467a-4b4e-d704: running image plugin create: pulling the image: unpacking layer `sha256:33db33d935721dfdf882423ee0e4d68410850032bc4e45779f7d746816710787`: failed to unpack: : waiting for the reexec command failed: unpacking-failed: opening target whiteout directory: open test: no such file or directory
15:22:15.270: [CELL.0] : exit status 1

or in another image:

Cell b87239a8-1b05-4ace-93b3-8a29484886d5 failed to create container for instance d6e31d31-7947-4ca7-7eda-651b: OCI runtime create failed: container_linux.go:346: starting container process caused "process_linux.go:449: container init caused \"rootfs_linux.go:58: mounting \\\"/var/vcap/data/garden/bin/init\\\" to rootfs \\\"/var/vcap/data/grootfs/store/unprivileged/images/d6e31d31-7947-4ca7-7eda-651b/rootfs\\\" at \\\"/var/vcap/data/grootfs/store/unprivileged/images/d6e31d31-7947-4ca7-7eda-651b/rootfs/tmp/garden-init\\\" caused \\\"open /var/vcap/data/grootfs/store/unprivileged/images/d6e31d31-7947-4ca7-7eda-651b/rootfs/tmp/garden-init: permission denied\\\"\"": unknown

Expected behavior
The expected behavior looks like this when building with Docker (docker build .) and starting with cf push (Dockerimage build with docker: thebohne/kaniko:docker):

15:49:29.176: [CELL.0] Cell f3551e37-379f-454a-81d3-389a4c6d12a1 successfully created container for instance 77902562-a6aa-4aaf-7659-d8b3
15:49:31.431: [APP/PROC/WEB.0] Exit status 143

To Reproduce
For you, I wrote a minimal Dockerfile:

FROM debian:stretch
RUN mkdir /test
RUN echo "TEST" > /test/testfile
RUN mv /test/testfile /usr/local/bin

The same error should occur with other base images as well.
I am running the following commands in my Jenkinspipeline (there should be no difference when running on your local maschine):

/kaniko/executor --destination=thebohne/kaniko:kaniko --context=${WORKSPACE} --dockerfile=src/main/docker/Dockerfile or with --cache=true
CF_DOCKER_PASSWORD=${DOCKER_PASSWORD} cf push ${NAME} --docker-image thebohne/kaniko:kaniko --docker-username ${USERNAME} -u process

Test Matrix
Here a little matrix of what I tested:

	Run with Docker	Run in Cloud Foundry
Build with Docker	✔️	✔️
Build with Kaniko	✔️	❌

Else
So I don't know where to post this issue (I hope I'm in the right repo 😁) because there's a link between Kaniko and Cloud Foundry.
Here is the link to the same issue that was published in the Kaniko repo: GoogleContainerTools/kaniko#1149

Thanks for you help :) if you need additional information please write a comment :)

Why does grootfs unpack the downloaded tar layer?

I'm trying to deploy bosh-lite. downloadLayer function tries to decompress the downloaded tar archive

however bosh-warden-cpi-release expects to see the archive and I get an error below:

Creating VM with agent ID '{{eb2931fc-1e68-44ef-8492-876b7f01647f}}': Creating container: running image plugin create: invalid base image: directory provided instead of a tar file

I tried to work this around by packing the directory back to the tar:

mv /var/vcap/store/warden_cpi/stemcells/06a29be3-73c4-4d61-5871-5a49a32f313d /var/vcap/store/warden_cpi/stemcells/06a29be3-73c4-4d61-5871-5a49a32f313d.dir
cd /var/vcap/store/warden_cpi/stemcells/06a29be3-73c4-4d61-5871-5a49a32f313d.dir
tar -cf ../06a29be3-73c4-4d61-5871-5a49a32f313d .

Did I miss a config option for warden or grootfs?

Gitbot test.

Please configure GITBOT

@claurence : Pivotal uses GITBOT to synchronize Github issues and pull requests with Pivotal Tracker.
Please add your new repo to the GITBOT config-prouction.yml in the Gitbot configuration repo
If there are any questions, please reach out to [email protected]

Feature Request: Support for "Content trust" for docker images

What?

Docker supports a feature called "Content trust" : https://docs.docker.com/engine/security/trust/content_trust/ where the docker images are verified against a signature when they are downloaded from the registry.

Some tenants are requesting us this feature, asking to enable this check per application.

Grootfs can validate the image as it gets downloaded. The challenge is how the user can enable this feature and how the rest of the cloudfoundry stack should handle the situation.

Maybe a custom parameter in the docker url? e.g. docker://myregistry/image?content_trust=true

	logger.Debug(fmt.Sprintf("attempt-get-blob-%d", i+1))
	blob, size, e := imgSrc.GetBlob(context.TODO(), blobInfo, none.NoCache)