carvel-dev / imgpkg Goto Github PK
View Code? Open in Web Editor NEWStore application configuration files in Docker/OCI registries
Home Page: https://carvel.dev/imgpkg
License: Apache License 2.0
Store application configuration files in Docker/OCI registries
Home Page: https://carvel.dev/imgpkg
License: Apache License 2.0
I wasn't sure where else to post this, but it looks like something similar is cooking up in this KEP: https://github.com/kubernetes/enhancements/pull/1481/files
we no longer use NewDigest to parse refs here: https://github.com/vmware-tanzu/carvel-imgpkg/blob/develop/pkg/imgpkg/cmd/image_set.go#L41-L46 (commend is out of date now). but unless something changed we should still be parsing refs as digests. if we are trying to transfer over tags then i would imagine it should be an additional field and not muddy up ref itself.
imgpkg tag list
features a --digests
option that is described as Include digests (default true)
. Since the option is true by default, it will always be true regardless of whether or not it is specified. This calls into question whether the flag should exist at all or whether it should be false by default.
Edit
Repurposing this issue around changing the default to false
for --digests
for imgpkg tag list
as documented in this comment.
In reviewing the workflow documentation for imgpkg, there is currently no mention of using a BundleLock in the workflows.
It would be helpful to document when someone should make use of a BundleLock file and try and find a way to incorporate it into the workflow docs. While the current documentation on the concept of the BundleLock is clear, it is not entirely clear when one should be used.
Vote on this request
This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.
👍 "I would like to see this bug fixed as soon as possible"
👎 "There are more important bugs to focus on right now"
We are also happy to receive and review Pull Requests if you want to help fix this issue.
ability to use env vars to set registry auth: https://github.com/k14s/kbld/blob/develop/docs/auth.md#via-environment-variables (https://github.com/k14s/kbld/blob/develop/pkg/kbld/registry/env_keychain.go)
We should more from ghdoss's yaml package to the k8s yaml package, which is a more maintained fork.
Using ggcr's verify closer allows us to ensure the bits we read match the expected hash
When storing configuration files in Git, it's easy to show diff between different versions. When storing configuration files in a docker image, we will need an imgpkg sub command to show the diff (which is similar to the --diff-changes of kapp)
Vote on this request
This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.
👍 "I would like to see this bug fixed as soon as possible"
👎 "There are more important bugs to focus on right now"
We are also happy to receive and review Pull Requests if you want to help fix this issue.
Describe the problem/challenge you have
When first creating a Bundle with imgpkg
, I would like the ability to generate all the contents of the .imgpkg
directory with a single command. I would like to use this command directly through imgpkg
and not directly use an additional tool such as kbld
.
Describe the solution you'd like
imgpkg init -f /files
creates the ImagesLock and Bundle metadata config in the .imgpkg
directory.
Where applicable, we should consider allowing the user to have options to fill in the properties of the bundle metadata file.
imgpkg init -f /files --name my-bundle
Anything else you would like to add:
The behavior in this case could be calling out to kbld
to generate the ImagesLock file as a sensible default.
The latest release v0.1.0 was published on Nov 2, 2019. There are several commits after then. Should we create a new release like 0.1.1 or 0.2.0? Thanks.
Describe the problem/challenge you have
we have several sources for copy command:
--bundle
--image
--lock
--from-tar
only one of them has "from".
and few destinations:
--to-repo
--to-tar
Describe the solution you'd like
either we add "from" to all sources or drop "from" from tar.
Does the error...
During the first rounds of user testing, we found that our proposed error messages had addressed questions 1-4, but lacked any guidance on how to fix the problem. Since we know that our proposed changes to the error messages are improvements (based on user tests), we think it would be smartest to implement these improvements iteratively, rather than waiting until we have the "perfect" error message.
During the user testing stage, we worked out a general template on how we want the new error messages to look, however we only drafted complete errors in three different scenarios.
Template Error Message:
ytt: Error: Overlaying (in order: <overlay file>):
Expected:
<overlay file>:<line failure> (<match type>, '<node value>') to match <expected match number> item in
<template file>:<line failure> (<match type> in '<parent of match>') but matched <actual match number>.
<overlay file>:<line failure>
<context from overlay file consisting of:
- document start (---)
- parent tree down to line of failure
- annotation matcher above the failure line
- ellipsis (...) to skip lines such as siblings and unimportant context
>
<template file>:<line failure>
<context from the template file consisting of:
- document start (---)
- parent tree down to lines where the overlay matched
- ellipsis (...) to skip lines such as siblings and unimportant context
>
Since an overlay can match via an expression, the domain of overlay error messages is vast, making it difficult to implement changes. In order to address this, a HackMD Doc was created with a subset of scenarios and proposed errors. This document will also act as the acceptance criteria for the following implementation story.
At this time, this document is not the source of truth, and we are looking identify more error scenarios. Once an example error message is fleshed out for each major category that is identified, there should be enough information for implementation to begin.
When running tests locally for imgpkg
, the TestCopyBundleLockInputToRepoUsingGCloudWithAnExpiredToken
test some times passes but more often fails. It however does not appear to fail in CI. So the best way to reproduce this would be to run tests locally using ./hack/test-all.sh
.
Failure logs shown below:
=== RUN TestCopyBundleLockInputToRepoUsingGCloudWithAnExpiredToken
Running 'imgpkg push -b index.docker.io/k8slt/imgpkg-test:1608583014597425000 -f /var/folders/mq/20zdxg1556jgl0h_p2744rc40000gq/T/imgpkg-test-copy-bundleLock-repo --lock-output /var/folders/mq/20zdxg1556jgl0h_p2744rc40000gq/T/imgpkg-test-copy-bundleLock-repo/bundle.lock.yml --yes'...
Running 'imgpkg copy --lock /var/folders/mq/20zdxg1556jgl0h_p2744rc40000gq/T/imgpkg-test-copy-bundleLock-repo/bundle.lock.yml --to-repo index.docker.io/k8slt/imgpkg-test-relocation --lock-output /var/folders/mq/20zdxg1556jgl0h_p2744rc40000gq/T/bundle-lock-relocate-lock.yml --yes'...
imgpkg.go:82: Failed to successfully execute 'imgpkg copy --lock /var/folders/mq/20zdxg1556jgl0h_p2744rc40000gq/T/imgpkg-test-copy-bundleLock-repo/bundle.lock.yml --to-repo index.docker.io/k8slt/imgpkg-test-relocation --lock-output /var/folders/mq/20zdxg1556jgl0h_p2744rc40000gq/T/bundle-lock-relocate-lock.yml --yes': Execution error: stdout: '' stderr: 'copy | exporting 4 images...
copy | will export gcr.io/cf-k8s-lifecycle-tooling-klt/kpack-build-init@sha256:8136ff3a64517457b91f86bf66b8ffe13b986aaf3511887eda107e59dcb8c632
copy | will export gcr.io/cf-k8s-lifecycle-tooling-klt/kpack-completion@sha256:1e83c4ccb56ad3e0fccbac74f91dfc404db280f8d3380cfa20c7d68fd0359235
copy | will export gcr.io/cf-k8s-lifecycle-tooling-klt/nginx@sha256:f35b49b1d18e083235015fd4bbeeabf6a49d9dc1d3a1f84b7df3794798b70c13
copy | will export index.docker.io/k8slt/imgpkg-test@sha256:d9ae7ae936dcbb29fb9b9a86c3b4dae04c29ab7b3372510046cdfb98c46df925
ERROR: gcloud crashed (OperationalError): database is locked
ERROR: gcloud crashed (OperationalError): database is locked
ERROR: gcloud crashed (OperationalError): database is locked
ERROR: gcloud crashed (OperationalError): database is locked
ERROR: gcloud crashed (OperationalError): database is locked
ERROR: gcloud crashed (OperationalError): database is locked
ERROR: gcloud crashed (OperationalError): database is locked
ERROR: gcloud crashed (OperationalError): database is locked
ERROR: gcloud crashed (OperationalError): database is locked
ERROR: gcloud crashed (OperationalError): database is locked
copy | exported 4 images
Error: Collecting packaging metadata: Working with gcr.io/cf-k8s-lifecycle-tooling-klt/kpack-completion@sha256:1e83c4ccb56ad3e0fccbac74f91dfc404db280f8d3380cfa20c7d68fd0359235: Retried 5 times: error getting credentials - err: exit status 1, out: ``
' error: 'exit status 1'
--- FAIL: TestCopyBundleLockInputToRepoUsingGCloudWithAnExpiredToken (49.42s)
What steps did you take:
$ bin/imgpkg push -b localhost:5000/projects-bundle:v1 -f bundle -f deployment
dir: .
dir: .imgpkg
file: .imgpkg/images.yml
dir: .
dir: config
file: config/app.yml
file: config/crds.yml
file: config/helpers.star
file: config/projects.yml
file: config/schema.yml
Pushed 'localhost:5000/projects-bundle@sha256:e19829cc0bd2e8713bfb5abbe899aac0f87c5d60639ed22b589b16d550f0174c'
Succeeded
$ bin/imgpkg copy -b localhost:5000/projects-bundle:v1 --to-repo k8slt/projects-bundle
copy | exporting 2 images...
copy | will export localhost:5000/projects-app@sha256:c46eb0c974a21b4cb3e7508125bc7fa7aabf7f4dc9d6321a083df5330192c656
copy | will export localhost:5000/projects-bundle@sha256:e19829cc0bd2e8713bfb5abbe899aac0f87c5d60639ed22b589b16d550f0174c
copy | exported 2 images
copy | importing 2 images...
copy | importing localhost:5000/projects-bundle@sha256:e19829cc0bd2e8713bfb5abbe899aac0f87c5d60639ed22b589b16d550f0174c -> index.docker.io/k8slt/projects-bundle@sha256:e19829cc0bd2e8713bfb5abbe899aac0f87c5d60639ed22b589b16d550f0174c...
copy | importing localhost:5000/projects-app@sha256:c46eb0c974a21b4cb3e7508125bc7fa7aabf7f4dc9d6321a083df5330192c656 -> index.docker.io/k8slt/projects-bundle@sha256:c46eb0c974a21b4cb3e7508125bc7fa7aabf7f4dc9d6321a083df5330192c656...
copy | imported 2 images
Succeeded
$ bin/imgpkg copy -b k8slt/projects-bundle@sha256:e19829cc0bd2e8713bfb5abbe899aac0f87c5d60639ed22b589b16d550f0174c --to-repo localhost:5000/projects-bundle-to-deploy
copy | exporting 2 images...
copy | will export index.docker.io/k8slt/projects-bundle@sha256:e19829cc0bd2e8713bfb5abbe899aac0f87c5d60639ed22b589b16d550f0174c
copy | will export localhost:5000/projects-app@sha256:c46eb0c974a21b4cb3e7508125bc7fa7aabf7f4dc9d6321a083df5330192c656
copy | exported 2 images
copy | importing 2 images...
copy | importing index.docker.io/k8slt/projects-bundle@sha256:e19829cc0bd2e8713bfb5abbe899aac0f87c5d60639ed22b589b16d550f0174c -> localhost:5000/projects-bundle-to-deploy@sha256:e19829cc0bd2e8713bfb5abbe899aac0f87c5d60639ed22b589b16d550f0174c...
copy | importing localhost:5000/projects-app@sha256:c46eb0c974a21b4cb3e7508125bc7fa7aabf7f4dc9d6321a083df5330192c656 -> localhost:5000/projects-bundle-to-deploy@sha256:c46eb0c974a21b4cb3e7508125bc7fa7aabf7f4dc9d6321a083df5330192c656...
copy | imported 2 images
Succeeded
What happened:
On step 3 images were copied from the original registry
What did you expect:
Expect images copied to come from the repository from step 2
Anything else you would like to add:
The problem is in https://github.com/vmware-tanzu/carvel-imgpkg/blob/develop/pkg/imgpkg/bundle/bundle_images_lock.go#L31
Looks like that we previously expected the imagesLock
that is provided in the constructor to be updated, but the LocalizeImagesLock
function creates a brand new copy of the lock file with the localized images
Environment:
imgpkg --version
): imgpkg version 0.4.0Docker HUB
): registry:2 image and docker hub/etc/os-release
): MacOSTasks:
When an overlay matcher’s expectation fails, the error message ytt
gives falls short of helping most users determine what to do next.
For more details see the "Problem" and "Analysis" sections in Overlay message errors
Do this loop until we find a good error message:
Note: By "little" we intend to put in the smallest possible effort that could still yield enough signal that we'll mitigate the risk we build the wrong thing. In other words, it's better to do this loop more times than it would to invest a lot in a single loop.
As part of #62 a 'workaround' solution was found to ggcr's inability to write a non-distributable layer.
The workaround solution was to write the 'foreign' layers upfront before trying to write the image manifest.
We decided to proceed with the workaround solution, since there was some urgency behind getting window images working. See https://kubernetes.slack.com/archives/CH8KCCKA5/p1598013615011900
Create a PR or Issue to https://github.com/google/go-containerregistry to allow writing a non-distributable layer.
Change imgpkg to use the above change to writing an image manifest that contains a 'foreign' image layer.
Allows us to have better package boundaries and import a small ImagesLock package in to kbld instead of duplicating the ImagesLock logic. This will help keep kbld and imgpkg in sync wrt imgpkg lock files.
Describe the problem/challenge you have
If I would like to transfer a bundle to another, private registry, I would also like the images to be extracted into the registry so they can be used in more contexts than just Kubernetes. For instance if I transfer a bundle to a private registry, all of my images are stored as sha tags on the bundle image. This makes it impossible to run these images in any context other than Kubernetes such as docker run
without first attaining the lock file and searching for the image you would like to run.
Take the case when I have an app bundle with simple-app
image in the bundle. I have moved this to my private registry with imgpkg copy
. I would now like to run docker run <private-registry>/simple-app
so I can debug the container. In order to do this I will need to first imgpkg pull
, inspect the lock file contents, find the correct sha value, and then docker run <private-registry>/my-bundle:sha@long-sha-value
Describe the solution you'd like
I would like to have an extract command to expand my bundle back to how it was when it was created. For instance imgpkg extract -b <private-registry, or tar file> --to-repo <private-registry>
This would then extract the captured images so I can run docker run <private-registry>/simple-app
As A Software Packager
I want to Create a bundle to distribute the multiple applications that are part of a deployment
So that Application consumers can retrieve all needed software from a single place
The following images exist in the registry
my.registry.io/bundle-1@sha256:aaaaaaaaaafdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
The .imgpkg/images.yml
looks like the following
apiVersion: imgpkg.carvel.dev/v1alpha1
kind: ImagesLock
images:
- image: my.registry.io/img-1@sha256:bbbbbbbbbbfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
- image: my.registry.io/bundle-2@sha256:ddddddddddfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
my.registry.io/bundle-2@sha256:ddddddddddfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
The .imgpkg/images.yml
looks like the following
apiVersion: imgpkg.carvel.dev/v1alpha1
kind: ImagesLock
images:
- image: my.registry.io/img-2@sha256:eeeeeeeeeefdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
my.registry.io/img-1@sha256:bbbbbbbbbbfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
my.registry.io/img-2@sha256:eeeeeeeeeefdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
--experimental-recursive-bundle
Given the following file exists in r-bundle/.imgpkg/images.yml
apiVersion: imgpkg.carvel.dev/v1alpha1
kind: ImagesLock
images:
- image: my.registry.io/bundle-1@sha256:aaaaaaaaaafdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
- image: my.registry.io/bundle-2@sha256:ddddddddddfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
When I run imgpkg push -b my.registry.io/r-bundle -f r-bundle/ --experimental-recursive-bundle
Then I should see
dir: r-bundle/
dir: r-bundle/.imgpkg
file: r-bundle/.imgpkg/images.yml
Pushed 'my.registry.io/r-bundle@sha256:ccccccccccfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0'
Succeeded
When I run imgpkg copy -b my.registry.io/r-bundle@sha256:ccccccccccfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0 --to-repo new.registry.io/new-bundle --experimental-recursive-bundle
Then I should see
copy | exporting 5 images...
copy | will export my.registry.io/img-1@sha256:bbbbbbbbbbfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
copy | will export my.registry.io/bundle-2@sha256:ddddddddddfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
copy | will export my.registry.io/img-2@sha256:eeeeeeeeeefdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
copy | will export my.registry.io/bundle-1@sha256:aaaaaaaaaafdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
copy | will export my.registry.io/r-bundle@sha256:ccccccccccfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
copy | exported 5 images
copy | importing 5 images...
copy | importing my.registry.io/img-1@sha256:bbbbbbbbbbfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0 -> new.registry.io/new-bundle@sha256:bbbbbbbbbbfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0...
copy | importing my.registry.io/bundle-2@sha256:ddddddddddfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0 -> new.registry.io/new-bundle@sha256:ddddddddddfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0...
copy | importing my.registry.io/img-2@sha256:eeeeeeeeeefdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0 -> new.registry.io/new-bundle@sha256:eeeeeeeeeefdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0...
copy | importing my.registry.io/bundle-1@sha256:aaaaaaaaaafdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0 -> new.registry.io/new-bundle@sha256:aaaaaaaaaafdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0...
copy | importing my.registry.io/r-bundle@sha256:ccccccccccfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0 -> new.registry.io/new-bundle@sha256:ccccccccccfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
copy | imported 5 images
Succeeded
--experimental-recursive-bundle
on simple bundlesGiven Pre-requirements
When I run imgpkg copy -b my.registry.io/bundle-2@sha256:ddddddddddfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0 --to-repo new.registry.io/new-bundle
Then I should see
copy | exporting 2 images...
copy | will export my.registry.io/img-2@sha256:eeeeeeeeeefdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
copy | will export my.registry.io/bundle-2@sha256:ddddddddddfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
copy | exported 2 images
copy | importing 2 images...
copy | importing my.registry.io/img-2@sha256:eeeeeeeeeefdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0 -> new.registry.io/new-bundle@sha256:eeeeeeeeeefdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0...
copy | importing my.registry.io/bundle-2@sha256:ddddddddddfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0 -> new.registry.io/new-bundle@sha256:ddddddddddfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0...
copy | imported 2 images
Succeeded
--experimental-recursive-bundle
on recursive bundle bundlesGiven the following file exists in r-bundle/.imgpkg/images.yml
apiVersion: imgpkg.carvel.dev/v1alpha1
kind: ImagesLock
images:
- image: my.registry.io/bundle-1@sha256:aaaaaaaaaafdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
- image: my.registry.io/bundle-2@sha256:ddddddddddfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
When I run imgpkg push -b my.registry.io/r-bundle -f r-bundle/ --experimental-recursive-bundle
Then I should see
dir: r-bundle/
dir: r-bundle/.imgpkg
file: r-bundle/.imgpkg/images.yml
Pushed 'my.registry.io/r-bundle@sha256:ccccccccccfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0'
Succeeded
When I run imgpkg copy -b my.registry.io/r-bundle@sha256:ccccccccccfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0 --to-repo new.registry.io/new-bundle
Then I should see
Error: This bundle contains bundles, in order to copy please execute the following command
imgpkg copy -b my.registry.io/r-bundle@sha256:ccccccccccfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0 --to-repo new.registry.io/new-bundle --experimental-recursive-bundle
Notes:
Describe the problem/challenge you have
Describe the solution you'd like
currently one would have to do the following:
this seems a bit repetitive. proposed steps:
i think having an explicit flag like --to-same-bundle is good to avoid confusion of now specifying anything for other from-to pairs.
Anything else you would like to add:
this "promotes" the use case for ensuring i have all materials for the bundle in my registry to first class.
this might be also useful to "repair" bundle in your registry if some references images got deleted.
Explore why imgpkg copy is inefficient.
The copy command takes a very long time to copy a simple image from a repository to a different one in the same OCI registry.
A small image like hello-world takes around 20 seconds to be copied.
I think this is tied to the network which lead us to think that there might be a download and upload of a layer that is already in the registry.
We should investigate if there is a more efficient way to do the copy of the layers.
For reference projects like pack does manipulation of layers and maybe there is some difference between what we are doing in our code and the way pack is pushing/pulling the images that it uses.
Describe the problem/challenge you have
Currently, imgpkg
can retain information about the tag of a bundle but when we try to use it with an image it does not retain it.
Describe the solution you'd like
The scenario of copying an image
Given a set of files in examples/basic-step-1
And create an image in the registry by executing imgpkg push -i localhost:5000/some-image:v0.5.0 -f examples/basic-step-1
When copying the image to a different repository by executing imgpkg copy -i localhost:5000/some-image:v0.5.0 --to-repo localhost:5000/some-other-image
Then I can execute docker pull localhost:5000/some-other-image:v0.5.0
And will be able to retrieve the image created in the first step
Based on a request from Zihan Jiang: Slack conversations
Describe the problem/challenge you have
we typically have not included spec
in various configuration resources in carvel. it feels a bit less consistent with other resources. i think we should drop it.
As A Software Packager
I want to Create a bundle to distribute the multiple applications that are part of a deployment
So that Application consumers can retrieve all needed software from a single place
Draft a proposal to address #10.
Publish the proposal on GitHub to gather feedback from the team and community.
Describe the problem/challenge you have
would like to see a vulnerability report for images that are part of a bundle.
Describe the solution you'd like
even though vulnerability scanning is an "addon" to a registry (not part of official registry APIs), i think we should try to integrate with projects like Harbor that introduce custom APIs to expose such information. i could imagine something like this:
$ imgpkg vulnreport -b registry.corp.com/apps/my-bundle
Id State Details Full reference
foo OK 0 vulnerabilities foo@sha256:...
blah FAIL 100 vulnerabilities foo@sha256:...
inspiration: carvel-dev/kbld#79
What steps did you take:
Create a bundle directory (bundle_dir) including the necessary .imgpkg dir and images.yml file
Attempt to push the bundle directory as an image
imgpkg push -i <repo> -f bundle_dir
Observe the expected error.
...consider using a bundle
Now, exclude the .imgpkg directory from the push command.
imgpkg push -i <repo> -f bundle_dir --file-exclude-defaults .imgpkg
What happened:
The same error.
...consider using a bundle
What did you expect:
The directory and its contents pushed as an image to the repo.
Unsure if this is a pathway we want to allow/encourage, it's a little strange to be pushing a bundle as an image, but I would expect the tool to respect my "exclude" choices.
Anything else you would like to add:
This happens because of where our excluded files are checked, only when making the tarball, after the image/bundle verification steps.
Environment:
imgpkg --version
): imgpkg version 0.2.0Docker HUB
): Any repo/etc/os-release
): Macwe currently do not allow to reference a bundle as an image inside a bundle. example where this might be helpful is: "we have a bundle that delivers configuration that references bunch of other bundles to be installed later".
gotta make sure that we dedup the tree since there could be cycles.
This is an epic placeholder. For more specific context, see #65.
What steps did you take:
When running imgpkg pull foo -o /tmp
, you will get the following error message:
Error: command 'foo' does not accept extra arguments 'imgpkg pull'
This also happens with imgpkg push foo
and imgpkg copy foo
:
Error: command 'foo' does not accept extra arguments 'imgpkg push'
Error: command 'foo' does not accept extra arguments 'imgpkg copy'
What did you expect:
An error stating that no arguments are accepted by these commands. Maybe also what options are necessary to use each command.
Anything else you would like to add:
The above error is from Cobra, but perhaps it makes sense to add our own validation/error with something like the following with imgpkg push
. This could be added to the command itself. An issue with this is that imgpkg
traditionally has its error messages displayed in red text, but the error comes out with no color.
PreRunE: func(cmd *cobra.Command, args []string) error {
if len(args) > 0 {
return fmt.Errorf("command 'imgpkg %s' accepts no args and requires the '--image' or '--bundle' with '--file' option", cmd.Name())
}
return nil
}
}
imgpkg (pronounced: image package) allows to store sets of files (e.g. application configuration) as images in Docker (OCI) registries.
If 'image package' means package an docker image, the name is good; if it means k8s application configuration package, then a name like 'configpkg' or 'konfigpkg' is better :D
Describe the problem/challenge you have
I'm symlinking files into my config directory, but when I try to package them with imgpkg I get:
error: Adding file 'config' to tar: Expected file 'config/eventing' to be a regular file
Describe the solution you'd like
I don't see why there needs to be a constraint preventing symlinks from working
As A Software Packager
I want to Create a bundle to distribute the multiple applications that are part of a deployment
So that Application consumers can retrieve all needed software from a single place
The following images exist in the registry
my.registry.io/bundle-1@sha256:aaaaaaaaaafdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
my.registry.io/img-1@sha256:bbbbbbbbbbfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
my.registry.io/img-2@sha256:ccccccccccfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
The .imgpkg/images.yml
looks like the following
apiVersion: imgpkg.carvel.dev/v1alpha1
kind: ImagesLock
images:
- image: my.registry.io/img-1@sha256:bbbbbbbbbbfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
Given the following file exists in r-bundle/.imgpkg/images.yml
apiVersion: imgpkg.carvel.dev/v1alpha1
kind: ImagesLock
images:
- image: my.registry.io/bundle-1@sha256:aaaaaaaaaafdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
When I run imgpkg push -b my.registry.io/r-bundle -f r-bundle/ --experimental-recursive-bundle
Then I should see
dir: r-bundle/
dir: r-bundle/.imgpkg
file: r-bundle/.imgpkg/images.yml
Pushed 'my.registry.io/r-bundle@sha256:ccccccccccfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0'
Succeeded
When I run imgpkg pull -b my.registry.io/r-bundle@sha256:ccccccccccfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0 -o /tmp/recursive-bundle
Then I should see
Pulling bundle 'my.registry.io/r-bundle@sha256:ccccccccccfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0'
Extracting layer 'sha256:ccccccccccfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0' (1/1)
Locating image lock file images...
One or more images not found in bundle repo; skipping lock file update
Succeeded
Given the following file exists in r-bundle/.imgpkg/images.yml
apiVersion: imgpkg.carvel.dev/v1alpha1
kind: ImagesLock
images:
- image: my.registry.io/bundle-1@sha256:aaaaaaaaaafdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
- image: my.registry.io/img-1@sha256:bbbbbbbbbbfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
When I run imgpkg push -b my.registry.io/r-bundle -f r-bundle/ --experimental-recursive-bundle
Then I should see
dir: r-bundle/
dir: r-bundle/.imgpkg
file: r-bundle/.imgpkg/images.yml
Pushed 'my.registry.io/r-bundle@sha256:ccccccccccfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0'
Succeeded
When I run imgpkg pull -b my.registry.io/r-bundle@sha256:ccccccccccfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0 -o /tmp/recursive-bundle
Then I should see
Pulling bundle 'my.registry.io/r-bundle@sha256:ccccccccccfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0'
Extracting layer 'sha256:ccccccccccfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0' (1/1)
Locating image lock file images...
One or more images not found in bundle repo; skipping lock file update
Succeeded
--experimental-recursive-bundle
flagGiven the following file exists in r-bundle/.imgpkg/images.yml
apiVersion: imgpkg.carvel.dev/v1alpha1
kind: ImagesLock
images:
- image: my.registry.io/bundle-1@sha256:aaaaaaaaaafdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
When I run imgpkg push -b my.registry.io/r-bundle -f r-bundle/
Then I should see
Error: Expected image lock to not contain bundle reference: 'my.registry.io/bundle-1@sha256:aaaaaaaaaafdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0'
Notes:
imgpkg pull
will not be changed in this story, a future story will change it to match the proposal--experimental-recursive-bundle
to pushThis issue was discussed in carvel-dev/kbld#75.
Since imgpkg
inherited some functionality from kbld
we will need to fix it in imgpkg
as well.
Notes
Describe the problem/challenge you have
For users managing manifests in a different repository than their main projects, there is no way with imgpkg push -f
to reference folders or files via a url.
Describe the solution you'd like
Support pushing folders/files via a url with imgpkg push -f
like below:
imgpkg push -f https://raw.githubusercontent.com/vmware-tanzu/carvel-imgpkg/9ecde093d13a203f9cbf844ede5951aa8d489c97/examples/basic-step-1/config.yml -i user/repo
Anything else you would like to add:
This could also assist with educating users about imgpkg
when they go through the imgpkg
workflow docs. Being able to reference remote files as opposed to needing to pull down examples while going through the workflow would be helpful for users just getting started.
Imgpkg should support a flag to allow the user to relocate images with foreign layers to airgapped envs.
Slack thread for some discussion: https://kubernetes.slack.com/archives/CH8KCCKA5/p1598013615011900
As A Software Packager
I want to Create a bundle to distribute the multiple applications that are part of a deployment
So that Application consumers can retrieve all needed software from a single place
The following images exist in the registry
my.registry.io/bundle-1@sha256:aaaaaaaaaafdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
The .imgpkg/images.yml
looks like the following
apiVersion: imgpkg.carvel.dev/v1alpha1
kind: ImagesLock
images:
- image: my.registry.io/img-1@sha256:bbbbbbbbbbfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
- image: my.registry.io/bundle-2@sha256:ddddddddddfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
my.registry.io/bundle-2@sha256:ddddddddddfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
The .imgpkg/images.yml
looks like the following
apiVersion: imgpkg.carvel.dev/v1alpha1
kind: ImagesLock
images:
- image: my.registry.io/img-2@sha256:ccccccccccfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
my.registry.io/img-1@sha256:bbbbbbbbbbfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
my.registry.io/img-2@sha256:ccccccccccfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
-r
flagGiven the following file exists in r-bundle/.imgpkg/images.yml
apiVersion: imgpkg.carvel.dev/v1alpha1
kind: ImagesLock
images:
- image: my.registry.io/bundle-1@sha256:aaaaaaaaaafdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
- image: my.registry.io/bundle-2@sha256:ddddddddddfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
When I run imgpkg push -b my.registry.io/r-bundle -f r-bundle/ --experimental-recursive-bundle
Then I should see
dir: r-bundle/
dir: r-bundle/.imgpkg
file: r-bundle/.imgpkg/images.yml
Pushed 'my.registry.io/r-bundle@sha256:ccccccccccfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0'
Succeeded
When I run imgpkg pull -r -b my.registry.io/r-bundle@sha256:ccccccccccfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0 -o /tmp/recursive-bundle
Then I should see
Pulling bundle 'my.registry.io/r-bundle@sha256:ccccccccccfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0'
Bundle Layers
Extracting layer 'sha256:87bf2c587b3315143cd05df7bd24d4e608ddb59f8c62110fe1b579fb817a2917' (1/1)
Nested bundles
Pulling Bundle 'my.registry.io/bundle-1@sha256:aaaaaaaaaafdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0' (1/2)
Extracting layer 'sha256:81fc6f37c9774541136e6113d899c215151496f4cf91c89c056783d2feb5ae0d' (1/1)
Found 1 Bundle packaged
Pulling Nested Bundle 'my.registry.io/bundle-2@sha256:ddddddddddfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0' (1/1)
Extracting layer 'sha256:9abb11371e7e53b5c33da086ea50dabb5d4cdd280be7d489169374b0188feab1' (1/1)
Pulling Nested Bundle 'my.registry.io/bundle-2@sha256:ddddddddddfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0' (2/2)
Skipped, already downloaded
Locating image lock file images...
One or more images not found in bundle repo; skipping lock file update
Succeeded
When I execute tree -a /tmp/recursive-bundle
Then I should see
/tmp/recursive-bundle
├── .imgpkg
| ├── bundles
| │ ├── sha256-ddddddddddfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
| │ │ ├── .imgpkg
| │ │ │ ├── bundle.yml
| │ │ │ └── images.yml
| │ │ └── config2.yml
| │ └── sha256-aaaaaaaaaafdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
| │ ├── .imgpkg
| │ │ ├── bundle.yml
| │ │ └── images.yml
| │ └── config1.yml
│ ├── bundle.yml
│ └── images.yml
└── config.yml
7 directories, 9 files
--recursive
flagNote: same as the above scenario but using the long version of the flag
Given the following file exists in r-bundle/.imgpkg/images.yml
apiVersion: imgpkg.carvel.dev/v1alpha1
kind: ImagesLock
images:
- image: my.registry.io/bundle-1@sha256:aaaaaaaaaafdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
- image: my.registry.io/bundle-2@sha256:ddddddddddfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
When I run imgpkg push -b my.registry.io/r-bundle -f r-bundle/ --experimental-recursive-bundle
Then I should see
dir: r-bundle/
dir: r-bundle/.imgpkg
file: r-bundle/.imgpkg/images.yml
Pushed 'my.registry.io/r-bundle@sha256:ccccccccccfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0'
Succeeded
When I run imgpkg pull --recursive -b my.registry.io/r-bundle@sha256:ccccccccccfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0 -o /tmp/recursive-bundle
Then I should see
Pulling bundle 'my.registry.io/r-bundle@sha256:ccccccccccfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0'
Bundle Layers
Extracting layer 'sha256:87bf2c587b3315143cd05df7bd24d4e608ddb59f8c62110fe1b579fb817a2917' (1/1)
Nested bundles
Pulling Bundle 'my.registry.io/bundle-1@sha256:aaaaaaaaaafdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0' (1/2)
Extracting layer 'sha256:81fc6f37c9774541136e6113d899c215151496f4cf91c89c056783d2feb5ae0d' (1/1)
Found 1 Bundle packaged
Pulling Nested Bundle 'my.registry.io/bundle-2@sha256:ddddddddddfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0' (1/1)
Extracting layer 'sha256:9abb11371e7e53b5c33da086ea50dabb5d4cdd280be7d489169374b0188feab1' (1/1)
Pulling Nested Bundle 'my.registry.io/bundle-2@sha256:ddddddddddfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0' (2/2)
Skipped, already downloaded
Locating image lock file images...
One or more images not found in bundle repo; skipping lock file update
Succeeded
When I execute tree -a /tmp/recursive-bundle
Then I should see
/tmp/recursive-bundle
├── .imgpkg
| ├── bundles
| │ ├── sha256-ddddddddddfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
| │ │ ├── .imgpkg
| │ │ │ ├── bundle.yml
| │ │ │ └── images.yml
| │ │ └── config2.yml
| │ └── sha256-aaaaaaaaaafdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
| │ ├── .imgpkg
| │ │ ├── bundle.yml
| │ │ └── images.yml
| │ └── config1.yml
│ ├── bundle.yml
│ └── images.yml
└── config.yml
7 directories, 9 files
-r
flagNote: There is a change in the output of the pull command
Given the following file exists in r-bundle/.imgpkg/images.yml
apiVersion: imgpkg.carvel.dev/v1alpha1
kind: ImagesLock
images:
- image: my.registry.io/bundle-1@sha256:aaaaaaaaaafdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
- image: my.registry.io/bundle-2@sha256:ddddddddddfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
When I run imgpkg push -b my.registry.io/r-bundle -f r-bundle/ --experimental-recursive-bundle
Then I should see
dir: r-bundle/
dir: r-bundle/.imgpkg
file: r-bundle/.imgpkg/images.yml
Pushed 'my.registry.io/r-bundle@sha256:ccccccccccfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0'
Succeeded
When I run imgpkg pull -b my.registry.io/r-bundle@sha256:ccccccccccfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0 -o /tmp/recursive-bundle
Then I should see
Pulling bundle 'my.registry.io/r-bundle@sha256:ccccccccccfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0'
Bundle Layers
Extracting layer 'sha256:87bf2c587b3315143cd05df7bd24d4e608ddb59f8c62110fe1b579fb817a2917' (1/1)
Nested bundles
Bundle 'my.registry.io/bundle-1@sha256:aaaaaaaaaafdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0' (1/2)
Bundle 'my.registry.io/bundle-2@sha256:ddddddddddfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0' (2/2)
To retrieve the configuration of all the nested bundles use the command
imgpkg pull --recursive -b my.registry.io/r-bundle@sha256:ccccccccccfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0 -o /tmp/recursive-bundle
Locating image lock file images...
One or more images not found in bundle repo; skipping lock file update
Succeeded
-r
flagGiven the pre requirements exist
When I run imgpkg pull -r -b my.registry.io/bundle-2@sha256:ddddddddddfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
Then I should see
Pulling bundle 'my.registry.io/r-bundle@sha256:ddddddddddfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0'
Bundle Layers
Extracting layer 'sha256:87bf2c587b3315143cd05df7bd24d4e608ddb59f8c62110fe1b579fb817a2917' (1/1)
Locating image lock file images...
One or more images not found in bundle repo; skipping lock file update
Succeeded
As part of imgpkg
refactoring (#24), issue #17 was closed. #17 describes moving ImagesLock/BundleLock into their own package. In #24, these lock files were moved to a package called lockfiles
.
This issue is being opened to document any changes needed to convert kbld
to using imgpkg
's lockfiles
package for using its ImageLock definition. kbld
's implementation of ImagesLock differs a bit, and we should figure out what needs to be done to convert the imgpkg
concept of an ImagesLock to more closely align and the ImagesLock defined in kbld
can be removed.
This issue can be closed when:
imgpkg
ImageLock definition to support kbld
(if any)kbld
can import imgpkg
's lockfiles
package for its own use cases with ImagesLocksthis is surprising as a user of e2e tests since changes to .docker/config.json affects my other operations on the computer. can we find an alternative way of testing whatever needs to be tested?
As A Software Packager
I want to Create a bundle to distribute the multiple applications that are part of a deployment
So that Application consumers can retrieve all needed software from a single place
The following images exist in the registry
my.registry.io/bundle-1@sha256:aaaaaaaaaafdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
The .imgpkg/images.yml
looks like the following
apiVersion: imgpkg.carvel.dev/v1alpha1
kind: ImagesLock
images:
- image: my.registry.io/img-1@sha256:bbbbbbbbbbfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
- image: my.registry.io/bundle-2@sha256:ddddddddddfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
my.registry.io/bundle-2@sha256:ddddddddddfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
The .imgpkg/images.yml
looks like the following
apiVersion: imgpkg.carvel.dev/v1alpha1
kind: ImagesLock
images:
- image: my.registry.io/img-2@sha256:eeeeeeeeeefdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
my.registry.io/img-1@sha256:bbbbbbbbbbfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
my.registry.io/img-2@sha256:eeeeeeeeeefdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
Given the following file exists in r-bundle/.imgpkg/images.yml
apiVersion: imgpkg.carvel.dev/v1alpha1
kind: ImagesLock
images:
- image: my.registry.io/bundle-1@sha256:aaaaaaaaaafdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
- image: my.registry.io/bundle-2@sha256:ddddddddddfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
When I run imgpkg push -b my.registry.io/r-bundle -f r-bundle/ --experimental-recursive-bundle
Then I should see
dir: r-bundle/
dir: r-bundle/.imgpkg
file: r-bundle/.imgpkg/images.yml
Pushed 'my.registry.io/r-bundle@sha256:ccccccccccfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0'
Succeeded
When I run imgpkg copy -b my.registry.io/r-bundle@sha256:ccccccccccfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0 --to-tar /tmp/recursive-bundle.tar --experimental-recursive-bundle
Then I should see
copy | exporting 5 images...
copy | will export my.registry.io/img-1@sha256:bbbbbbbbbbfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
copy | will export my.registry.io/bundle-2@sha256:ddddddddddfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
copy | will export my.registry.io/r-bundle@sha256:ccccccccccfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
copy | will export my.registry.io/bundle-1@sha256:aaaaaaaaaafdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
copy | will export my.registry.io/img-2@sha256:eeeeeeeeeefdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
copy | exported 5 images
copy | writing layers...
copy | done: file 'manifest.json' (35.107µs)
copy | done: file 'sha256-81fc6f37c9774541136e6113d899c215151496f4cf91c89c056783d2feb5ae0d.tar.gz' (46.817µs)
copy | done: file 'sha256-9abb11371e7e53b5c33da086ea50dabb5d4cdd280be7d489169374b0188feab1.tar.gz' (99.194µs)
copy | done: file 'sha256-87bf2c587b3315143cd05df7bd24d4e608ddb59f8c62110fe1b579fb817a2917.tar.gz' (119.416µs)
copy | done: file 'sha256-8ece9ac45f2b7228b2ed95e9f407b4f0dc2ac74f93c62ff1156f24c53042ba54.tar.gz' (370.872834ms)
copy | done: file 'sha256-7ekwjlfkjiekserwefwdfwd95e9f407b4f0dc2ac74f93c62ff1156f24c530sdfs43.tar.gz' (130.39ms)
Succeeded
When I execute tar xvfz /tmp/recursive-bundle.tar -C /tmp/rec-bundle && tree /tmp/rec-bundle
Then I should see
/tmp/rec-bundle
├── manifest.json
├── sha256-81fc6f37c9774541136e6113d899c215151496f4cf91c89c056783d2feb5ae0d.tar.gz
├── sha256-9abb11371e7e53b5c33da086ea50dabb5d4cdd280be7d489169374b0188feab1.tar.gz
├── sha256-87bf2c587b3315143cd05df7bd24d4e608ddb59f8c62110fe1b579fb817a2917.tar.gz
├── sha256-8ece9ac45f2b7228b2ed95e9f407b4f0dc2ac74f93c62ff1156f24c53042ba54.tar.gz
└── sha256-7ekwjlfkjiekserwefwdfwd95e9f407b4f0dc2ac74f93c62ff1156f24c530sdfs43.tar.gz
6 files
Notes:
Given Scenario 1 tar file
When I run imgpkg copy --tar /tmp/recursive-bundle.tar --to-repo new.registry.io/new-bundle
Then I should see
copy | importing 5 images...
copy | importing my.registry.io/img-1@sha256:bbbbbbbbbbfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0 -> new.registry.io/new-bundle@sha256:bbbbbbbbbbfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0...
copy | importing my.registry.io/bundle-2@sha256:ddddddddddfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0 -> new.registry.io/new-bundle@sha256:ddddddddddfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0...
copy | importing my.registry.io/img-2@sha256:eeeeeeeeeefdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0 -> new.registry.io/new-bundle@sha256:eeeeeeeeeefdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0...
copy | importing my.registry.io/bundle-1@sha256:aaaaaaaaaafdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0 -> new.registry.io/new-bundle@sha256:aaaaaaaaaafdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0...
copy | importing my.registry.io/r-bundle@sha256:ccccccccccfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0 -> new.registry.io/new-bundle@sha256:ccccccccccfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0...
copy | imported 5 images
Succeeded
Notes:
--experimental-recursive-bundle
on recursive bundle bundlesGiven the following file exists in r-bundle/.imgpkg/images.yml
apiVersion: imgpkg.carvel.dev/v1alpha1
kind: ImagesLock
images:
- image: my.registry.io/bundle-1@sha256:aaaaaaaaaafdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
- image: my.registry.io/bundle-2@sha256:ddddddddddfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
When I run imgpkg push -b my.registry.io/r-bundle -f r-bundle/ --experimental-recursive-bundle
Then I should see
dir: r-bundle/
dir: r-bundle/.imgpkg
file: r-bundle/.imgpkg/images.yml
Pushed 'my.registry.io/r-bundle@sha256:ccccccccccfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0'
Succeeded
When I run imgpkg copy -b my.registry.io/r-bundle@sha256:ccccccccccfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0 --to-tar /tmp/recursive-bundle.tar
Then I should see
Error: This bundle contains bundles, in order to copy please execute the following command
imgpkg copy -b my.registry.io/r-bundle@sha256:ccccccccccfdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0 --to-tar /tmp/recursive-bundle.tar --experimental-recursive-bundle
Got this:
$ imgpkg copy -b k8slt/webservers-demo:0.0.1 --to-tar=the-football.tar
copy | exporting 2 images...
copy | will export index.docker.io/k8slt/webservers-demo@sha256:5ebbd85028685b3f343fb6736109441643f2f85f59c919774c02d29943185477
copy | will export index.docker.io/library/nginx@sha256:b4b9b3eee194703fc2fa8afa5b7510c77ae70cfba567af1376a573a967c03dbb
copy | exported 2 images
copy | writing layers...
copy | done: file 'manifest.json' (40.388µs)
copy | done: file 'sha256-01a0caaddf4b559a9603a60e9cdd072d8b405d7c78a0a83a92e834b3c0898733.tar.gz' (9.126µs)
copy | done: file 'sha256-0a4690c5d889e116874bf45dc757b515565a3bd9b0f6c04054d62280bb4f4ecf.tar.gz' (14.790296ms)
copy | done: file 'sha256-0e75755515daa100dcb232b335d5b61d0278b5850bf0992f2927447031769a3e.tar.gz' (15.836402ms)
copy | done: file 'sha256-2eead4197fac409644fd8aaf115559d6383b0d56f1ad04d7116aaabbcbea8bed.tar.gz' (36.950171ms)
copy | done: file 'sha256-44446b456159d28726ce1ce5eaf894fa98415dce50d113ffdcc6cb99505bd109.tar.gz' (5.527µs)
copy | done: file 'sha256-458117d4230ebcca9a649824e2a9a8d8a7c3b39ee0cca6ae57cc11e0debe713e.tar.gz' (3.863µs)
copy | done: file 'sha256-45d6b85619b1cdde5c938d82a688d73c5532d0d9992b1af33d4a47aa09e0b699.tar.gz' (20.615259ms)
copy | done: file 'sha256-494350645e1c39e92e069b34b7ce0310dc6496d163f078bb1400391a3a574105.tar.gz' (27.39677ms)
copy | done: file 'sha256-55541b6f4770186b6d3e005cd06a9508301ed3c92877d4e627faed87923ffdbc.tar.gz' (4.93µs)
copy | done: file 'sha256-5e1d4775cd524fabdf3b94fcf2a517eb62c74ff55ccf7c32779218df94026295.tar.gz' (3.844µs)
copy | done: file 'sha256-7f05216ff691c8948e167b003b68728fe14854394d4f1287224705f9487985e4.tar.gz' (26.332055ms)
copy | done: file 'sha256-9719afee3eb7680a805449b50040fed1fff0d065f58d186e6e2bb4959f3d9e6e.tar.gz' (34.64653ms)
copy | done: file 'sha256-989ec9c2d5c0faa3c30e41cf0fbdb98632d8d5ef12aa71801d619454a65a3df3.tar.gz' (21.506775ms)
copy | done: file 'sha256-9a806ca4c22e0f67425c88867deec63e29dbd1463d60e4bf4ec32ef8af0fa174.tar.gz' (5.423µs)
copy | done: file 'sha256-af822a1ba456a0c6c2c0e954103b6b630ca93b8ea69cd8c34496d0149c94b27b.tar.gz' (24.455231ms)
copy | done: file 'sha256-b6857afb67f0f43f41d622a660c1164719c1622b528284dd739a4b35e7bb07e4.tar.gz' (4.368µs)
copy | done: file 'sha256-d8f0c6f8a424139159764e11d49d9ad098fc72ebb8397f31b6d26e13545f9a04.tar.gz' (24.878672ms)
copy | done: file 'sha256-f01e2fc672b937f58be589455802ade04da5bae6a188a46a6090252055118d90.tar.gz' (36.67724ms)
copy | done: file 'sha256-fa05dcee3c0c86d4f7c7198c7083a2d314b6bfa3fb483e9c6d80317c5d7fe09d.tar.gz' (41.844033ms)
copy | done: file 'sha256-44446b456159d28726ce1ce5eaf894fa98415dce50d113ffdcc6cb99505bd109.tar.gz' (64.148µs)
copy | done: file 'sha256-d8f0c6f8a424139159764e11d49d9ad098fc72ebb8397f31b6d26e13545f9a04.tar.gz' (8.689763164s)
copy | done: file 'sha256-f01e2fc672b937f58be589455802ade04da5bae6a188a46a6090252055118d90.tar.gz' (7.20779871s)
copy | done: file 'sha256-458117d4230ebcca9a649824e2a9a8d8a7c3b39ee0cca6ae57cc11e0debe713e.tar.gz' (28.163µs)
copy | done: file 'sha256-494350645e1c39e92e069b34b7ce0310dc6496d163f078bb1400391a3a574105.tar.gz' (2.572949236s)
copy | done: file 'sha256-b6857afb67f0f43f41d622a660c1164719c1622b528284dd739a4b35e7bb07e4.tar.gz' (30.823µs)
copy | done: file 'sha256-01a0caaddf4b559a9603a60e9cdd072d8b405d7c78a0a83a92e834b3c0898733.tar.gz' (81.487µs)
copy | done: file 'sha256-af822a1ba456a0c6c2c0e954103b6b630ca93b8ea69cd8c34496d0149c94b27b.tar.gz' (15.296891022s)
copy | done: file 'sha256-fa05dcee3c0c86d4f7c7198c7083a2d314b6bfa3fb483e9c6d80317c5d7fe09d.tar.gz' (6.293347155s)
copy | done: file 'sha256-55541b6f4770186b6d3e005cd06a9508301ed3c92877d4e627faed87923ffdbc.tar.gz' (33.065µs)
copy | done: file 'sha256-0a4690c5d889e116874bf45dc757b515565a3bd9b0f6c04054d62280bb4f4ecf.tar.gz' (17.881268948s)
copy | done: file 'sha256-9a806ca4c22e0f67425c88867deec63e29dbd1463d60e4bf4ec32ef8af0fa174.tar.gz' (29.568µs)
copy | done: file 'sha256-7f05216ff691c8948e167b003b68728fe14854394d4f1287224705f9487985e4.tar.gz' (4.82642281s)
copy | done: file 'sha256-9719afee3eb7680a805449b50040fed1fff0d065f58d186e6e2bb4959f3d9e6e.tar.gz' (22.618497646s)
copy | done: file 'sha256-5e1d4775cd524fabdf3b94fcf2a517eb62c74ff55ccf7c32779218df94026295.tar.gz' (31.008µs)
copy | done: file 'sha256-989ec9c2d5c0faa3c30e41cf0fbdb98632d8d5ef12aa71801d619454a65a3df3.tar.gz' (5.397030992s)
copy | done: file 'sha256-0e75755515daa100dcb232b335d5b61d0278b5850bf0992f2927447031769a3e.tar.gz' (6.610824467s)
copy | done: file 'sha256-2eead4197fac409644fd8aaf115559d6383b0d56f1ad04d7116aaabbcbea8bed.tar.gz' (5.568175778s)
copy | done: file 'sha256-45d6b85619b1cdde5c938d82a688d73c5532d0d9992b1af33d4a47aa09e0b699.tar.gz' (5.627179799s)
when
$ tar tvf the-football.tar | awk '{print $9}' | sort
manifest.json
sha256-01a0caaddf4b559a9603a60e9cdd072d8b405d7c78a0a83a92e834b3c0898733.tar.gz
sha256-0a4690c5d889e116874bf45dc757b515565a3bd9b0f6c04054d62280bb4f4ecf.tar.gz
sha256-0e75755515daa100dcb232b335d5b61d0278b5850bf0992f2927447031769a3e.tar.gz
sha256-2eead4197fac409644fd8aaf115559d6383b0d56f1ad04d7116aaabbcbea8bed.tar.gz
sha256-44446b456159d28726ce1ce5eaf894fa98415dce50d113ffdcc6cb99505bd109.tar.gz
sha256-458117d4230ebcca9a649824e2a9a8d8a7c3b39ee0cca6ae57cc11e0debe713e.tar.gz
sha256-45d6b85619b1cdde5c938d82a688d73c5532d0d9992b1af33d4a47aa09e0b699.tar.gz
sha256-494350645e1c39e92e069b34b7ce0310dc6496d163f078bb1400391a3a574105.tar.gz
sha256-55541b6f4770186b6d3e005cd06a9508301ed3c92877d4e627faed87923ffdbc.tar.gz
sha256-5e1d4775cd524fabdf3b94fcf2a517eb62c74ff55ccf7c32779218df94026295.tar.gz
sha256-7f05216ff691c8948e167b003b68728fe14854394d4f1287224705f9487985e4.tar.gz
sha256-9719afee3eb7680a805449b50040fed1fff0d065f58d186e6e2bb4959f3d9e6e.tar.gz
sha256-989ec9c2d5c0faa3c30e41cf0fbdb98632d8d5ef12aa71801d619454a65a3df3.tar.gz
sha256-9a806ca4c22e0f67425c88867deec63e29dbd1463d60e4bf4ec32ef8af0fa174.tar.gz
sha256-af822a1ba456a0c6c2c0e954103b6b630ca93b8ea69cd8c34496d0149c94b27b.tar.gz
sha256-b6857afb67f0f43f41d622a660c1164719c1622b528284dd739a4b35e7bb07e4.tar.gz
sha256-d8f0c6f8a424139159764e11d49d9ad098fc72ebb8397f31b6d26e13545f9a04.tar.gz
sha256-f01e2fc672b937f58be589455802ade04da5bae6a188a46a6090252055118d90.tar.gz
sha256-fa05dcee3c0c86d4f7c7198c7083a2d314b6bfa3fb483e9c6d80317c5d7fe09d.tar.gz
Is there a way to either mate-up each "copy from" with its "copy to"? or clarify the source/sink involved in a given line of output?
What steps did you take:
@DennisDenuto and I ran imgpkg
in an internetless environment. We copied a tar file to a repository:
./imgpkg copy -i ubuntu --to-tar ubuntu.tar
./imgpkg-linux-amd64 copy --tar ./ubuntu.tar --to-repo air.gapped.registry/image
What happened:
imgpkg tries to get the image in the tar from the original registry
copy | importing 1 images...
copy | importing index.docker.io/library/ubuntu@sha256:3093096ee188f8ff4531949b8f6115af4747ec1c58858c091c8cb4579c39cc4e -> 172.19.0.2/rept/image@sha256:3093096ee188f8ff4531949b8f6115af4747ec1c58858c091c8cb4579c39cc4e...
copy | imported 0 images
Error: Importing image index.docker.io/library/ubuntu@sha256:3093096ee188f8ff4531949b8f6115af4747ec1c58858c091c8cb4579c39cc4e: Importing image as 172.19.0.2/rept/image@sha256:3093096ee188f8ff4531949b8f6115af4747ec1c58858c091c8cb4579c39cc4e: Writing image: Retried 5 times: Get "https://172.19.0.2/v2/": dial tcp 172.19.0.2:443: connect: connection refused; Get "http://172.19.0.2/v2/": dial tcp 172.19.0.2:80: connect: connection refused
What did you expect:
I expect that imgpkg
copies the image from the tar file, not from index.docker.io/library/ubuntu
.
Anything else you would like to add:
[Additional information that will assist in solving the issue.]
Environment:
imgpkg --version
): imgpkg version 0.4.0Docker HUB
): docker/etc/os-release
): linuxThis is an epic placeholder. The context for this work can be found in #8
What steps did you take:
kbld -f examples/simple-app-build-and-push/ --imgpkg-lock-output foo.yml
imgpkg copy --lock foo.yml --lock-output foo2.yml --to-repo localhost:5000/foo
---
apiVersion: imgpkg.carvel.dev/v1alpha1
images:
- annotations:
kbld.carvel.dev/id: simple-app
image: localhost:5000/cppforlife/kbld-simple-app2@sha256:09f45eea0d452749445d1d015745b2bc5cfcbfb49f952d482d14f37ae7680a38
kind: ImagesLock
---
apiVersion: imgpkg.carvel.dev/v1alpha1
images:
- image: localhost:5000/foo@sha256:09f45eea0d452749445d1d015745b2bc5cfcbfb49f952d482d14f37ae7680a38
kind: ImagesLock
What happened:
The annotations
were dropped when I copied the image
What did you expect:
They'd be maintained, so that the mappings of app -> image would be maintained
Environment:
imgpkg --version
): 0.3.0Describe the problem/challenge you have
[A description of the current challenge that you are experiencing.]
Currently I can't specify a tag when using imgpkg copy -i example/image:tag --to-repo repo.example.com/my/image
so I need to add the tag after looking up the digest and applying it manually
Describe the solution you'd like
[A clear and concise description of what you want to happen. If applicable a visual representation of the UX (ex: new CLI argument name, the behavior expected).]
Either support a :tag
suffix for the --to-repo
arg or have a separate --tag
option
Anything else you would like to add:
[Additional information that will assist in solving the issue.]
Also the error message when I included the tag isn't the most helpful:
Error: Building import repository ref: repository can only contain the runes `abcdefghijklmnopqrstuvwxyz0123456789_-./`: trisberg/my-image:my-tag
Vote on this request
This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.
👍 "I would like to see this bug fixed as soon as possible"
👎 "There are more important bugs to focus on right now"
We are also happy to receive and review Pull Requests if you want to help fix this issue.
Describe the problem/challenge you have
In a scenario where the artifact that will be shared is a tar, there might not be a need to use the registry as an intermediary step. This would allow the users to combine the commands imgpkg push
+ imgpkg copy --to-tar
in a single command and without storing extra images in a registry
Describe the solution you'd like
Scenario for images
Given a set of files in examples/basic-step-1
When execute imgpkg build --tag v0.5.0 -f examples/basic-step-1 --to-tar some.tar
Then copy the image to a registryimgpkg copy --tar some.tar --to-repo localhost:5000/some-image
And retrieve it using imgpkg pull -i localhost:5000/some-image:v0.5.0
Scenario for bundles
Given a bundle in examples/basic-step-2
When execute imgpkg build --tag v0.5.0 -f examples/basic-step-1 --to-tar some-bundle.tar
Then copy the bundle to a registryimgpkg copy --tar some-bundle.tar --to-repo localhost:5000/some-bundle
And retrieve it using imgpkg pull -b localhost:5000/some-image:v0.5.0
Notes:
Based on a request from Zihan Jiang: Slack conversations
As A Software Packager
I want to Create a bundle to distribute the multiple applications that are part of a deployment
So that Application consumers can retrieve all needed software from a single place
As a operator wishing to collect a bundle that includes software that includes non-distributable bits (e.g. Windows-based products) for use in an air-gapped scenario
I want to be able to override the default behavior of skipping transferring so-called non-distributable layers ("foreign layers" in the docker manifests, "non-distributable" in OCI manifests)
and I want the bits of those layers to be pulled and included in the tarball of the bundle
and I want those bits to be loaded into the registry I copy the tarballed bundle.
Original GitHub Issue: imgpkg#8
Requested by the .NET Build team (Malini Valliath, PM) multiple times since this summer.
PCS/MAPBU leadership has pointed to the importance of servicing our Windows-using customers — Ajay pointed out the importance of Windows support during an all-hands meeting, noting that a significant percentage of our customers use Windows.
Note: we say, "foreign layers", here. We wish for all this to be true for any non-distributable layer.
Given a bundle which includes a reference to an image containing a foreign layer (e.g. a bundle that references Microsoft's nanoserver — mcr.microsoft.com/windows/nanoserver:1809-amd64
)
When I copy that bundle to a tarball
And I indicate that non-distributable layers (including foreign layers) should be included
$ imgpkg copy -b index.docker.io/k8slt/bundle-with-foreign-layers --to-tar the-football.tar --include-non-distributable
Then the tarball includes the contents of the layer
Given a tarball containing a bundle which includes one or more foreign layers
And a registry enabled to receive foreign layers (
When I copy the bundle to an image reference (i.e. a location in a registry)
And I indicate that non-distributable layers (including foreign layers) should be included
$ imgpkg copy --from-tar the-football.tar --to-repo gcr.io/cf-k8s-lifecycle-tooling-klt/bundle-with-foreign-layers --include-non-distributable
Then I can find that bundle in the registry AND its digest has not changed
And I can find those foreign layers in the registry
Given a bundle which includes a reference to an image containing a non-distributable layer (e.g. a foreign layer)
When I copy the bundle with no additional instruction to a tarball
$ imgpkg copy -b <image ref to bundle> --to-tar the-football.tar
Then I see a warning next to each non-distributable layer that indicates "skipped: foreign layer" or "skipped: non-distributable" (depending on the exact media type), similar to how Docker does it
And I see a warning in the summary that notes that one or more layers are not in the tarball because they are "non-distributable/foreign layers" and a link to documentation on how to include them, if desired
And I see that the overall operation is "Successful (with warnings)."
And the tarball does NOT include those layers (neither in the tarball itself, nor in the manifest.json
)
Given a tarball containing a bundle which includes one or more foreign layers
When I copy the bundle to an image reference (i.e. a location in a registry), with no additional instruction
$ imgpkg copy --from-tar the-football.tar --to-repo gcr.io/cf-k8s-lifecycle-tooling-klt/bundle-with-foreign-layers
Then I see a warning next to each non-distributable layer that indicates "skipped: foreign layer" or "skipped: non-distributable" (depending on the exact media type), similar to how Docker does it
And I see a warning in the summary that notes that one or more layers are not in the tarball because they are "non-distributable/foreign layers" and a link to documentation on how to include them, if desired
And I see that the overall operation is "Successful (with caveats)."
And I can NOT find those foreign layers in the registry
Given I want to know more about how imgpkg
handles non-distributed/foreign layers
When I read the documentation around the copy
command
Then I discover/understand:
imgpkg
respects the non-distributable nature of foreign layersapplication/vnd.oci.image.layer.nondistributable.v1.tar
or application/vnd.oci.image.layer.nondistributable.v1.tar+gzip
application/vnd.docker.image.rootfs.foreign.diff.tar.gzip
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.