Giter VIP home page Giter VIP logo

imagebuilder's Introduction

OCI Image Builder

Go Report Card GoDoc Travis Join the chat at freenode:openshift-dev

Please test your images (and add to our conformance suite)!

This library supports using the Dockerfile syntax to build OCI & Docker compatible images, without invoking a container build command such as buildah bud or docker build. It is intended to give clients more control over how they build container images, including:

  • Instead of building one layer per line, run all instructions in the same container
  • Set HostConfig settings like network and memory controls that are not available when running container builds
  • Mount external files into the build that are not persisted as part of the final image (i.e. "secrets")
  • If there are no RUN commands in the Dockerfile, the container is created and committed, but never started.

The final image should be 99.9% compatible with regular container builds, but bugs are always possible.

Future goals include:

  • Output OCI compatible images
  • Support other container execution engines, like runc or rkt
  • Better conformance testing
  • Windows support

Install and Run

To download and install the library and the binary, set up a Golang build environment and with GOPATH set run:

$ go install github.com/openshift/imagebuilder/cmd/imagebuilder@latest

The included command line takes one argument, a path to a directory containing a Dockerfile. The -t option can be used to specify an image to tag as:

$ imagebuilder [-t TAG] DIRECTORY

To mount a file into the image for build that will not be present in the final output image, run:

$ imagebuilder --mount ~/secrets/private.key:/etc/keys/private.key path/to/my/code testimage

Any processes in the Dockerfile will have access to /etc/keys/private.key, but that file will not be part of the committed image.

You can also customize which Dockerfile is run, or run multiple Dockerfiles in sequence (the FROM is ignored on later files):

$ imagebuilder -f Dockerfile:Dockerfile.extra .

will build the current directory and combine the first Dockerfile with the second. The FROM in the second image is ignored.

Note that imagebuilder adds the built image to the docker daemon's internal storage. If you use podman you must first pull the image into its local registry:

$ podman pull docker-daemon:<IMAGE>:<TAG> # must contain either a tag or a digest

Code Example

f, err := os.Open("path/to/Dockerfile")
if err != nil {
	return err
}
defer f.Close()

e := builder.NewClientExecutor(o.Client)
e.Out, e.ErrOut = os.Stdout, os.Stderr
e.AllowPull = true
e.Directory = "context/directory"
e.Tag = "name/of-image:and-tag"
e.AuthFn = nil // ... pass a function to retrieve authorization info
e.LogFn = func(format string, args ...interface{}) {
	fmt.Fprintf(e.ErrOut, "--> %s\n", fmt.Sprintf(format, args...))
}

buildErr := e.Build(f, map[string]string{"arg1":"value1"})
if err := e.Cleanup(); err != nil {
	fmt.Fprintf(e.ErrOut, "error: Unable to clean up build: %v\n", err)
}

return buildErr

Example of usage from OpenShift's experimental dockerbuild command with mount secrets

Run conformance tests (very slow):

docker rmi busybox; docker pull busybox
docker rmi centos:7; docker pull centos:7
chmod -R go-w ./dockerclient/testdata
go test ./dockerclient -tags conformance -timeout 30m

imagebuilder's People

Contributors

adambkaplan avatar ar01 avatar bertinatto avatar bparees avatar carbonin avatar dependabot[bot] avatar flouthoc avatar giuseppe avatar grooverdan avatar jfroy avatar jhult avatar jmguzik avatar jpeeler avatar mfojtik avatar mtrmac avatar mumoshu avatar nalind avatar openshift-ci[bot] avatar openshift-merge-bot[bot] avatar openshift-merge-robot avatar qiwang19 avatar quasilyte avatar rhatdan avatar saschagrunert avatar smarterclayton avatar sosiouxme avatar stevekuznetsov avatar tomsweeneyredhat avatar vdemeester avatar vrothberg avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

imagebuilder's Issues

wishlist: support for `--label` cmdline option

The docker build CLI has support for appending additional dynamic LABEL directives to the Dockerfile at build time and is often used to add (e.g.,) git remote / ref and other build information to the images. Currently imagebuilder doesn't appear to have a way of specifying these on the CLI

$ docker build --help

Usage:  docker build [OPTIONS] PATH | URL | -

Build an image from a Dockerfile

Options:
...
--label list              Set metadata for an image
...

Support for build timeouts

I'm hitting an intermittent hanging issue related to cross-building docker images, obviously I need to dig deeper into the cause of the hangs, but I think there could be value in adding support for timeouts to imagebuilder.

I'd be happy to submit a PR given a bit of guidance.

My initial thought was to add a flag to specify the desired timeout and then use that value in place of the hard coded 86400 sleep that is the default command used for the build image. It wasn't clear to me when the default value is not used, and the resulting error wouldn't be clear that it was related to the timeout expiring. Would it make more sense to handle the timeout separately and then gracefully clean up the build container after the timeout expires?

ADD --chown UID:GID dan.tar

Docker build now supports a chown operator on the ADD Function.

cat Dockerfile
FROM fedora
COPY --chown 3267:3267 dan.tar
CMD /bin/bash

COPY dir/filename-* /some_dir/some_filename resulted in some_filename being created as a directory when the source wildcard expansion results in a single file

The COPY command implementation on image builder doesn't seem to look at the result of a source wildcard expansion to determine whether the target should be a single file or a directory. Instead, it seems like the moment it sees the source as a wild-carded filename, it assumes the target is a dir. This is inconsistent with how other builders are implementing the COPY command.

Other builders such as podman, docker, or buildah will do the wild card expansion to determine if the target should be a file or a directory.

Error processing tar file with symlinks

When I'm trying to build images from https://github.com/openshift/oc or https://github.com/openshift/cluster-kube-apiserver-operator I'm getting:

$ make images
imagebuilder --allow-pull -t registry.svc.ci.openshift.org/ocp/4.2:cluster-kube-apiserver-operator -f ./Dockerfile.rhel7 .
--> FROM registry.svc.ci.openshift.org/ocp/builder:golang-1.12 as builder
--> WORKDIR /go/src/github.com/openshift/cluster-kube-apiserver-operator
--> COPY . .
API error (500): Error processing tar file(exit status 1): invalid symlink "/go/src/github.com/openshift/cluster-kube-apiserver-operator/vendor/github.com/coreos/etcd/Documentation/README.md" -> "docs.md"
make: *** [Makefile:20: image-ocp-cluster-kube-apiserver-operator] Error 1

Removing that particular link give me a similar error about a different link. Looks like there's a problem with symlinks.

tag a release

I am currently going through Podman's dependencies (updating the vendor.conf) trying to pin them to releases when possible and available. Some distributions, Debian in particular, are very conservative regarding go packaging as many libraries don't have any release (or no semantic versioning).

Can we tag a new release?

Plans re: containers/image and containers/storage support?

Hi! I've spent a bit of time recently working on a tool (currently http://github.com/nalind/buildah - be kind, it's early days) for building OCI images using https://github.com/containers/storage and https://github.com/containers/image, and before I go too far down the implementation road, I wanted to ask if you think if these things which buildah is looking to do would make more sense to implement as functions of imagebuilder:

  • using containers/storage for directly manipulating a working container and containers/image for importing and committing images, as an alternative to or in place of talking to dockerd
  • implementing instructions that would be found in a Dockerfile or equivalent as CLI tools that can be called from a command prompt or as part of a shell script, storing state in locations that the storage library offers, eventually allowing the image building process to be a shell interpreting a script instead of a single tool interpreting a Dockerfile

I expect the first bullet point could make things pretty messy, since it would likely require entirely different code paths, and the second may be completely out of scope for imagebuilder, but I'd like to know what the maintainers think.

VOLUME contents lose owner bits

Docker cp drops owner attributes of content during the UploadToContainer call, preferring the parent permissions. Injecting a static binary that can apply tar contents to disk correctly is probably the easiest option. That binary would have to be implemented per OS.

imagebuilder waits for child processes to complete; docker build doesn't

With a Dockerfile that runs a simple bash script which spawns a child, I found that docker build would finish and kill the build container as soon as the script finished, while imagebuilder would wait for the child to finish:

$ git clone https://github.com/sosiouxme/scratch -b 20180720-subprocess-wait

$ docker build --no-cache scratch
Sending build context to Docker daemon 72.19 kB
Step 1/3 : FROM fedora:28
 ---> 75aeb7f897fd
Step 2/3 : ADD script.sh /usr/bin/
 ---> 12ff6dbff87c
Removing intermediate container 64a93a2cbce6
Step 3/3 : RUN script.sh
 ---> Running in fae77c14c89b
parent done
 ---> ab6d22ef095c
Removing intermediate container fae77c14c89b
Successfully built ab6d22ef095c

$ imagebuilder scratch
--> FROM fedora:28 as 0
--> ADD script.sh /usr/bin/
--> RUN script.sh
parent done
child done
--> Committing changes ...
--> Done

This might not seem like a big problem but it is the source of my hung builds.

In the logging-kibana build, as part of the build, the node process starts up a daemon. This is because it runs some slow optimize/cache code up front, to store the result in the image so that it doesn't have to be done at runtime. But it doesn't properly kill the daemon so imagebuilder waits forever for it to finish.

Arguably this is just a badly-behaving build, just like if it started an infinite loop in the foreground. But infinite loops are more evident than forgotten children. If folks are testing their builds with docker they will not consider this a problem and imagebuilder will seem broken. And in our production OSBS, at least, the build never completes, blocking any other attempts at building until an admin can cancel it.

Incorrect output for '%' in strings

This Dockerfile:

FROM busybox
RUN stat -c %u /

produces this output:

--> FROM busybox
--> RUN stat -c %!u(MISSING) /
0
--> Committing changes ...
--> Done

It is only the output of which instruction is to be run that is incorrect. The correct instruction is run.

`go get` fails

$ go get -u github.com/openshift/imagebuilder/cmd/imagebuilder
# github.com/openshift/imagebuilder/vendor/github.com/docker/engine-api/types/filters
../../../go/src/github.com/openshift/imagebuilder/vendor/github.com/docker/engine-api/types/filters/parse.go:12:45: error: import file โ€˜github.com/docker/engine-api/types/versionsโ€™ not found
  "github.com/docker/engine-api/types/versions"
                                             ^
../../../go/src/github.com/openshift/imagebuilder/vendor/github.com/docker/engine-api/types/filters/parse.go:84:22: error: reference to undefined name โ€˜versionsโ€™
  if version != "" && versions.LessThan(version, "1.22") {
                      ^
# github.com/openshift/imagebuilder/vendor/github.com/docker/engine-api/types/container
../../../go/src/github.com/openshift/imagebuilder/vendor/github.com/docker/engine-api/types/container/config.go:4:45: error: import file โ€˜github.com/docker/engine-api/types/strsliceโ€™ not found
  "github.com/docker/engine-api/types/strslice"
                                             ^
../../../go/src/github.com/openshift/imagebuilder/vendor/github.com/docker/engine-api/types/container/config.go:5:38: error: import file โ€˜github.com/docker/go-connections/natโ€™ not found
  "github.com/docker/go-connections/nat"
                                      ^
../../../go/src/github.com/openshift/imagebuilder/vendor/github.com/docker/engine-api/types/container/config.go:21:25: error: expected package
  ExposedPorts    map[nat.Port]struct{} `json:",omitempty"` // List of exposed ports
                         ^
../../../go/src/github.com/openshift/imagebuilder/vendor/github.com/docker/engine-api/types/container/config.go:26:26: error: expected package
  Cmd             strslice.StrSlice     // Command to run when starting the container
                          ^
../../../go/src/github.com/openshift/imagebuilder/vendor/github.com/docker/engine-api/types/container/config.go:31:26: error: expected package
  Entrypoint      strslice.StrSlice     // Entrypoint to run when starting the container
                          ^
../../../go/src/github.com/openshift/imagebuilder/vendor/github.com/docker/engine-api/types/container/host_config.go:6:45: error: import file โ€˜github.com/docker/engine-api/types/blkiodevโ€™ not found
  "github.com/docker/engine-api/types/blkiodev"
                                             ^
../../../go/src/github.com/openshift/imagebuilder/vendor/github.com/docker/engine-api/types/container/host_config.go:7:45: error: import file โ€˜github.com/docker/engine-api/types/strsliceโ€™ not found
  "github.com/docker/engine-api/types/strslice"
                                             ^
../../../go/src/github.com/openshift/imagebuilder/vendor/github.com/docker/engine-api/types/container/host_config.go:8:38: error: import file โ€˜github.com/docker/go-connections/natโ€™ not found
  "github.com/docker/go-connections/nat"
                                      ^
../../../go/src/github.com/openshift/imagebuilder/vendor/github.com/docker/engine-api/types/container/host_config.go:9:28: error: import file โ€˜github.com/docker/go-unitsโ€™ not found
  "github.com/docker/go-units"
                            ^
../../../go/src/github.com/openshift/imagebuilder/vendor/github.com/docker/engine-api/types/container/host_config.go:221:34: error: expected package
  BlkioWeightDevice    []*blkiodev.WeightDevice
                                  ^
../../../go/src/github.com/openshift/imagebuilder/vendor/github.com/docker/engine-api/types/container/host_config.go:222:34: error: expected package
  BlkioDeviceReadBps   []*blkiodev.ThrottleDevice
                                  ^
../../../go/src/github.com/openshift/imagebuilder/vendor/github.com/docker/engine-api/types/container/host_config.go:223:34: error: expected package
  BlkioDeviceWriteBps  []*blkiodev.ThrottleDevice
                                  ^
../../../go/src/github.com/openshift/imagebuilder/vendor/github.com/docker/engine-api/types/container/host_config.go:224:34: error: expected package
  BlkioDeviceReadIOps  []*blkiodev.ThrottleDevice
                                  ^
../../../go/src/github.com/openshift/imagebuilder/vendor/github.com/docker/engine-api/types/container/host_config.go:225:34: error: expected package
  BlkioDeviceWriteIOps []*blkiodev.ThrottleDevice
                                  ^
../../../go/src/github.com/openshift/imagebuilder/vendor/github.com/docker/engine-api/types/container/host_config.go:238:31: error: expected package
  Ulimits              []*units.Ulimit // List of ulimits to be set in the container
                               ^
../../../go/src/github.com/openshift/imagebuilder/vendor/github.com/docker/engine-api/types/container/host_config.go:265:21: error: expected package
  PortBindings    nat.PortMap   // Port mapping between the exposed port (container) and the host
                     ^
../../../go/src/github.com/openshift/imagebuilder/vendor/github.com/docker/engine-api/types/container/host_config.go:272:26: error: expected package
  CapAdd          strslice.StrSlice // List of kernel capabilities to add to the container
                          ^
../../../go/src/github.com/openshift/imagebuilder/vendor/github.com/docker/engine-api/types/container/host_config.go:273:26: error: expected package
  CapDrop         strslice.StrSlice // List of kernel capabilities to remove from the container
                          ^
# github.com/openshift/imagebuilder/vendor/github.com/fsouza/go-dockerclient/external/github.com/docker/docker/pkg/system
../../../go/src/github.com/openshift/imagebuilder/vendor/github.com/fsouza/go-dockerclient/external/github.com/docker/docker/pkg/system/meminfo_linux.go:10:71: error: import file โ€˜github.com/fsouza/go-dockerclient/external/github.com/docker/go-unitsโ€™ not found
  "github.com/fsouza/go-dockerclient/external/github.com/docker/go-units"
                                                                       ^
../../../go/src/github.com/openshift/imagebuilder/vendor/github.com/fsouza/go-dockerclient/external/github.com/docker/docker/pkg/system/meminfo_linux.go:45:26: error: reference to undefined name โ€˜unitsโ€™
   bytes := int64(size) * units.KiB
                          ^
# github.com/openshift/imagebuilder/vendor/github.com/fsouza/go-dockerclient/external/github.com/docker/docker/pkg/ioutils
../../../go/src/github.com/openshift/imagebuilder/vendor/github.com/fsouza/go-dockerclient/external/github.com/docker/docker/pkg/ioutils/readers.go:8:69: error: import file โ€˜github.com/fsouza/go-dockerclient/external/golang.org/x/net/contextโ€™ not found
  "github.com/fsouza/go-dockerclient/external/golang.org/x/net/context"
                                                                     ^
../../../go/src/github.com/openshift/imagebuilder/vendor/github.com/fsouza/go-dockerclient/external/github.com/docker/docker/pkg/ioutils/readers.go:98:37: error: expected package
 func NewCancelReadCloser(ctx context.Context, in io.ReadCloser) io.ReadCloser {
                                     ^
../../../go/src/github.com/openshift/imagebuilder/vendor/github.com/fsouza/go-dockerclient/external/github.com/docker/docker/pkg/ioutils/readers.go:102:21: error: reference to undefined name โ€˜contextโ€™
  doneCtx, cancel := context.WithCancel(context.Background())
                     ^
../../../go/src/github.com/openshift/imagebuilder/vendor/github.com/fsouza/go-dockerclient/external/github.com/docker/docker/pkg/ioutils/readers.go:102:40: error: reference to undefined name โ€˜contextโ€™
  doneCtx, cancel := context.WithCancel(context.Background())
                                        ^
../../../go/src/github.com/openshift/imagebuilder/vendor/github.com/fsouza/go-dockerclient/external/github.com/docker/docker/pkg/ioutils/readers.go:113:13: error: expected channel
   case <-ctx.Done():
             ^
../../../go/src/github.com/openshift/imagebuilder/vendor/github.com/fsouza/go-dockerclient/external/github.com/docker/docker/pkg/ioutils/readers.go:125:14: error: expected channel
    case <-ctx.Done():
              ^
../../../go/src/github.com/openshift/imagebuilder/vendor/github.com/fsouza/go-dockerclient/external/github.com/docker/docker/pkg/ioutils/readers.go:127:18: error: expected channel
    case <-doneCtx.Done():
                  ^
# github.com/openshift/imagebuilder/vendor/github.com/docker/docker/builder/parser
../../../go/src/github.com/openshift/imagebuilder/vendor/github.com/docker/docker/builder/parser/parser.go:11:42: error: import file โ€˜github.com/docker/docker/builder/commandโ€™ not found
  "github.com/docker/docker/builder/command"
                                          ^
../../../go/src/github.com/openshift/imagebuilder/vendor/github.com/docker/docker/builder/parser/parser.go:50:3: error: reference to undefined name โ€˜commandโ€™
   command.User:       parseString,
   ^
../../../go/src/github.com/openshift/imagebuilder/vendor/github.com/docker/docker/builder/parser/parser.go:51:3: error: reference to undefined name โ€˜commandโ€™
   command.Onbuild:    parseSubCommand,
   ^
../../../go/src/github.com/openshift/imagebuilder/vendor/github.com/docker/docker/builder/parser/parser.go:52:3: error: reference to undefined name โ€˜commandโ€™
   command.Workdir:    parseString,
   ^
../../../go/src/github.com/openshift/imagebuilder/vendor/github.com/docker/docker/builder/parser/parser.go:53:3: error: reference to undefined name โ€˜commandโ€™
   command.Env:        parseEnv,
   ^
../../../go/src/github.com/openshift/imagebuilder/vendor/github.com/docker/docker/builder/parser/parser.go:54:3: error: reference to undefined name โ€˜commandโ€™
   command.Label:      parseLabel,
   ^
../../../go/src/github.com/openshift/imagebuilder/vendor/github.com/docker/docker/builder/parser/parser.go:55:3: error: reference to undefined name โ€˜commandโ€™
   command.Maintainer: parseString,
   ^
../../../go/src/github.com/openshift/imagebuilder/vendor/github.com/docker/docker/builder/parser/parser.go:56:3: error: reference to undefined name โ€˜commandโ€™
   command.From:       parseString,
   ^
../../../go/src/github.com/openshift/imagebuilder/vendor/github.com/docker/docker/builder/parser/parser.go:57:3: error: reference to undefined name โ€˜commandโ€™
   command.Add:        parseMaybeJSONToList,
   ^
../../../go/src/github.com/openshift/imagebuilder/vendor/github.com/docker/docker/builder/parser/parser.go:58:3: error: reference to undefined name โ€˜commandโ€™
   command.Copy:       parseMaybeJSONToList,
   ^
../../../go/src/github.com/openshift/imagebuilder/vendor/github.com/docker/docker/builder/parser/parser.go:59:3: error: reference to undefined name โ€˜commandโ€™
   command.Run:        parseMaybeJSON,
   ^
../../../go/src/github.com/openshift/imagebuilder/vendor/github.com/docker/docker/builder/parser/parser.go:60:3: error: reference to undefined name โ€˜commandโ€™
   command.Cmd:        parseMaybeJSON,
   ^
../../../go/src/github.com/openshift/imagebuilder/vendor/github.com/docker/docker/builder/parser/parser.go:61:3: error: reference to undefined name โ€˜commandโ€™
   command.Entrypoint: parseMaybeJSON,
   ^
../../../go/src/github.com/openshift/imagebuilder/vendor/github.com/docker/docker/builder/parser/parser.go:62:3: error: reference to undefined name โ€˜commandโ€™
   command.Expose:     parseStringsWhitespaceDelimited,
   ^
../../../go/src/github.com/openshift/imagebuilder/vendor/github.com/docker/docker/builder/parser/parser.go:63:3: error: reference to undefined name โ€˜commandโ€™
   command.Volume:     parseMaybeJSONToList,
   ^
# github.com/openshift/imagebuilder/vendor/github.com/fsouza/go-dockerclient/external/github.com/docker/docker/pkg/fileutils
../../../go/src/github.com/openshift/imagebuilder/vendor/github.com/fsouza/go-dockerclient/external/github.com/docker/docker/pkg/fileutils/fileutils.go:13:71: error: import file โ€˜github.com/fsouza/go-dockerclient/external/github.com/Sirupsen/logrusโ€™ not found
  "github.com/fsouza/go-dockerclient/external/github.com/Sirupsen/logrus"
                                                                       ^
../../../go/src/github.com/openshift/imagebuilder/vendor/github.com/fsouza/go-dockerclient/external/github.com/docker/docker/pkg/fileutils/fileutils_unix.go:10:71: error: import file โ€˜github.com/fsouza/go-dockerclient/external/github.com/Sirupsen/logrusโ€™ not found
  "github.com/fsouza/go-dockerclient/external/github.com/Sirupsen/logrus"
                                                                       ^
../../../go/src/github.com/openshift/imagebuilder/vendor/github.com/fsouza/go-dockerclient/external/github.com/docker/docker/pkg/fileutils/fileutils.go:116:3: error: reference to undefined name โ€˜logrusโ€™
   logrus.Debugf("Skipping excluded path: %s", file)
   ^
../../../go/src/github.com/openshift/imagebuilder/vendor/github.com/fsouza/go-dockerclient/external/github.com/docker/docker/pkg/fileutils/fileutils_unix.go:17:3: error: reference to undefined name โ€˜logrusโ€™
   logrus.Errorf("Error opening /proc/%d/fd: %s", os.Getpid(), err)
   ^
# github.com/openshift/imagebuilder/vendor/github.com/fsouza/go-dockerclient/external/github.com/docker/docker/pkg/stdcopy
../../../go/src/github.com/openshift/imagebuilder/vendor/github.com/fsouza/go-dockerclient/external/github.com/docker/docker/pkg/stdcopy/stdcopy.go:8:71: error: import file โ€˜github.com/fsouza/go-dockerclient/external/github.com/Sirupsen/logrusโ€™ not found
  "github.com/fsouza/go-dockerclient/external/github.com/Sirupsen/logrus"
                                                                       ^
../../../go/src/github.com/openshift/imagebuilder/vendor/github.com/fsouza/go-dockerclient/external/github.com/docker/docker/pkg/stdcopy/stdcopy.go:101:6: error: reference to undefined name โ€˜logrusโ€™
      logrus.Debugf("Corrupted prefix: %v", buf[:nr])
      ^
../../../go/src/github.com/openshift/imagebuilder/vendor/github.com/fsouza/go-dockerclient/external/github.com/docker/docker/pkg/stdcopy/stdcopy.go:107:5: error: reference to undefined name โ€˜logrusโ€™
     logrus.Debugf("Error reading header: %s", er)
     ^
../../../go/src/github.com/openshift/imagebuilder/vendor/github.com/fsouza/go-dockerclient/external/github.com/docker/docker/pkg/stdcopy/stdcopy.go:123:4: error: reference to undefined name โ€˜logrusโ€™
    logrus.Debugf("Error selecting output fd: (%d)", buf[stdWriterFdIndex])
    ^
../../../go/src/github.com/openshift/imagebuilder/vendor/github.com/fsouza/go-dockerclient/external/github.com/docker/docker/pkg/stdcopy/stdcopy.go:129:3: error: reference to undefined name โ€˜logrusโ€™
   logrus.Debugf("framesize: %d", frameSize)
   ^
../../../go/src/github.com/openshift/imagebuilder/vendor/github.com/fsouza/go-dockerclient/external/github.com/docker/docker/pkg/stdcopy/stdcopy.go:134:4: error: reference to undefined name โ€˜logrusโ€™
    logrus.Debugf("Extending buffer cap by %d (was %d)", frameSize+stdWriterPrefixLen-bufLen+1, len(buf))
    ^
../../../go/src/github.com/openshift/imagebuilder/vendor/github.com/fsouza/go-dockerclient/external/github.com/docker/docker/pkg/stdcopy/stdcopy.go:146:6: error: reference to undefined name โ€˜logrusโ€™
      logrus.Debugf("Corrupted frame: %v", buf[stdWriterPrefixLen:nr])
      ^
../../../go/src/github.com/openshift/imagebuilder/vendor/github.com/fsouza/go-dockerclient/external/github.com/docker/docker/pkg/stdcopy/stdcopy.go:152:5: error: reference to undefined name โ€˜logrusโ€™
     logrus.Debugf("Error reading frame: %s", er)
     ^
../../../go/src/github.com/openshift/imagebuilder/vendor/github.com/fsouza/go-dockerclient/external/github.com/docker/docker/pkg/stdcopy/stdcopy.go:160:4: error: reference to undefined name โ€˜logrusโ€™
    logrus.Debugf("Error writing frame: %s", ew)
    ^
../../../go/src/github.com/openshift/imagebuilder/vendor/github.com/fsouza/go-dockerclient/external/github.com/docker/docker/pkg/stdcopy/stdcopy.go:165:4: error: reference to undefined name โ€˜logrusโ€™
    logrus.Debugf("Error Short Write: (%d on %d)", nw, frameSize)
    ^
# github.com/openshift/imagebuilder/vendor/github.com/fsouza/go-dockerclient/external/github.com/docker/docker/pkg/homedir
../../../go/src/github.com/openshift/imagebuilder/vendor/github.com/fsouza/go-dockerclient/external/github.com/docker/docker/pkg/homedir/homedir.go:7:93: error: import file โ€˜github.com/fsouza/go-dockerclient/external/github.com/opencontainers/runc/libcontainer/userโ€™ not found
  "github.com/fsouza/go-dockerclient/external/github.com/opencontainers/runc/libcontainer/user"
                                                                                             ^
../../../go/src/github.com/openshift/imagebuilder/vendor/github.com/fsouza/go-dockerclient/external/github.com/docker/docker/pkg/homedir/homedir.go:25:16: error: reference to undefined name โ€˜userโ€™
   if u, err := user.CurrentUser(); err == nil {
                ^

Whiteout files not supported

When a whiteout file is created it should result in a file being removed from the resulting fs. For example:

$ cd $(mktemp -d)
$ echo "repo contents" > foo.repo
$ touch .wh.foo.repo
$ cat <<DF > Dockerfile
FROM fedora:28

ADD foo.repo /opt/foo.repo
RUN cat /opt/foo.repo
ADD .wh.foo.repo /opt/
RUN cat /opt/foo.repo
DF
$ docker build .
Sending build context to Docker daemon 3.584 kB
Step 1/5 : FROM fedora:28
 ---> cc510acfcd70
Step 2/5 : ADD foo.repo /opt/foo.repo
 ---> Using cache
 ---> fe5009f4a07d
Step 3/5 : RUN cat /opt/foo.repo
 ---> Using cache
 ---> 282b0d8f9bd2
Step 4/5 : ADD .wh.foo.repo /opt/
 ---> Using cache
 ---> 6d064295b23e
Step 5/5 : RUN cat /opt/foo.repo
 ---> Running in 9a82b576101a
cat: /opt/foo.repo: No such file or directory
The command '/bin/sh -c cat /opt/foo.repo' returned a non-zero code: 1

The command fails because the whiteout file takes effect. With imagebuilder in contrast:

$ imagebuilder -t foo .
--> FROM fedora:28 as 0
--> ADD foo.repo /opt/foo.repo
--> RUN cat /opt/foo.repo
repo contents
--> ADD .wh.foo.repo /opt/
--> RUN cat /opt/foo.repo
repo contents
--> Committing changes to foo ...
--> Done
[tmp.grxZVQQgyo] $ docker run foo ls -la /opt/
total 4
drwxr-xr-x. 1 root root 22 May 24 18:03 .
drwxr-xr-x. 1 root root  6 May 24 18:04 ..
-rw-rw-r--. 1 root root 14 May 24 17:21 foo.repo

The file remains. This is not really a difference in docker build, it's simply that whiteout files act on previous layers and with imagebuilder the whiteout file is placed in the same layer as the file it was intended to affect.

It's arguable whether we really want imagebuilder to emulate this somehow, but it would really simplify this use case.

SHELL env var

I'm guessing this is an issue, sorry if it isn't, but just a quick peruse through https://github.com/openshift/imagebuilder/pull/62/files I didn't see this.

In docker it is enough to just add ENV SHELL=/bin/bash to change the shell that is used. The SHELL keyword was added later, but I noticed prior to #62 ENV SHELL=/bin/bash is not supported and I'm guessing #62 doesn't add support for it.

Build fails: undefined: unix.Major and unix.Minor

Trying to build on Mac and running into:

$ pwd
/Users/lvlcek/go/src/github.com/openshift

$ go get github.com/openshift/imagebuilder
# github.com/openshift/imagebuilder/vendor/github.com/fsouza/go-dockerclient/internal/archive
imagebuilder/vendor/github.com/fsouza/go-dockerclient/internal/archive/archive_unix.go:71:25: undefined: unix.Major
imagebuilder/vendor/github.com/fsouza/go-dockerclient/internal/archive/archive_unix.go:72:25: undefined: unix.Minor

$ go version
go version go1.11.4 darwin/amd64

$ uname
Darwin

Imagebuilder is not handling multi line labels correctly

(Migrated from containers/podman#1149)

When building an image from a Dockerfile that has key=value pairs split across multiple lines of a single LABEL instruction, the resulting image has a mangled LABEL output.

This was observed on Fedora 28, using podman 0.7.2:

$ cat Dockerfile.multi 
FROM registry.fedoraproject.org/fedora:28
LABEL io.openshift.commit = "deadbeef" \
      io.openshift.version = "3.11"
ENTRYPOINT "/bin/bash"

$ sudo podman build -t quay.io/miabbott/podman-label-test:multi -f Dockerfile.multi
STEP 1: FROM registry.fedoraproject.org/fedora:28
STEP 2: LABEL io.openshift.commit = "deadbeef"       io.openshift.version = "3.11"
STEP 3: ENTRYPOINT "/bin/bash"
STEP 4: COMMIT containers-storage:[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.override_kernel_check=true]quay.io/miabbott/podman-label-test:multi
Getting image source signatures
Skipping fetch of repeat blob sha256:4656e146fab7f95ad90808bab5aa27e0041f1fa2427597a68efc6fa66693689d
Skipping fetch of repeat blob sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
Copying config sha256:2a84c16f3f8d0cead85069105deb6399e6d0d27a657b730d90e8e2abb8c71fa1
 733 B / 733 B [============================================================] 0s
Writing manifest to image destination
Storing signatures
--> 2a84c16f3f8d0cead85069105deb6399e6d0d27a657b730d90e8e2abb8c71fa1

$ sudo podman inspect quay.io/miabbott/podman-label-test:multi
[
    {
        "Id": "2a84c16f3f8d0cead85069105deb6399e6d0d27a657b730d90e8e2abb8c71fa1",
        "Digest": "sha256:d1c1fdaf1387d124cfdca4bd38b759c72177e2d72847c7f552b2bd7d383ddd5a",
        "RepoTags": [
            "quay.io/miabbott/podman-label-test:multi"
        ],
        "RepoDigests": [
            "quay.io/miabbott/podman-label-test@sha256:d1c1fdaf1387d124cfdca4bd38b759c72177e2d72847c7f552b2bd7d383ddd5a"                                                                                          
        ],
        "Parent": "",
        "Comment": "Created by Image Factory",
        "Created": "2018-07-24T14:12:44.280970058Z",
        "ContainerConfig": {
            "Env": [
                "DISTTAG=f28container",
                "FGC=f28",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],
            "Entrypoint": [
                "/bin/sh",
                "-c",
                "\"/bin/bash\""
            ],
            "WorkingDir": "/",
            "Labels": {
                "io.openshift.commit": "= deadbeef       io.openshift.version = 3.11",
                "license": "MIT",
                "name": "fedora",
                "vendor": "Fedora Project",
                "version": "28"
            }
        },
        "Version": "",
        "Author": "",
        "Architecture": "amd64",
        "Os": "linux",
        "Size": 262873295,
        "VirtualSize": 262873295,
        "GraphDriver": {
            "Name": "overlay",
            "Data": {
                "LowerDir": "/var/lib/containers/storage/overlay/4656e146fab7f95ad90808bab5aa27e0041f1fa2427597a68efc6fa66693689d/diff",
                "MergedDir": "/var/lib/containers/storage/overlay/091bcce0127a47ee3e4342cbc6fe065b2a7f0787718ce7dab7e755031df09902/merged",
                "UpperDir": "/var/lib/containers/storage/overlay/091bcce0127a47ee3e4342cbc6fe065b2a7f0787718ce7dab7e755031df09902/diff",
                "WorkDir": "/var/lib/containers/storage/overlay/091bcce0127a47ee3e4342cbc6fe065b2a7f0787718ce7dab7e755031df09902/work"
            }
        },
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:4656e146fab7f95ad90808bab5aa27e0041f1fa2427597a68efc6fa66693689d",
                "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef"
            ]
        },
        "Labels": {
            "io.openshift.commit": "= deadbeef       io.openshift.version = 3.11",
            "license": "MIT",
            "name": "fedora",
            "vendor": "Fedora Project",
            "version": "28"
        },
        "Annotations": {},
        "ManifestType": "application/vnd.oci.image.manifest.v1+json",
        "User": ""
    }
]

If I change the Dockerfile to use a single LABEL instruction per key=value pair, it works correctly:

$ cat Dockerfile.single 
FROM registry.fedoraproject.org/fedora:28
LABEL io.openshift.commit = "deadbeef"
LABEL io.openshift.version = "3.11"
ENTRYPOINT "/bin/bash"

$ sudo podman build -t quay.io/miabbott/podman-label-test:single -f Dockerfile.single
STEP 1: FROM registry.fedoraproject.org/fedora:28
STEP 2: LABEL io.openshift.commit = "deadbeef"
STEP 3: LABEL io.openshift.version = "3.11"
STEP 4: ENTRYPOINT "/bin/bash"
STEP 5: COMMIT containers-storage:[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.override_kernel_check=true]quay.io/miabbott/podman-label-test:single
Getting image source signatures
Skipping fetch of repeat blob sha256:4656e146fab7f95ad90808bab5aa27e0041f1fa2427597a68efc6fa66693689d
Skipping fetch of repeat blob sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
Copying config sha256:b05f036107d096578f7ced852104d3a9506079f77cbc3c03cbcbc9b88d4d8f27
 729 B / 729 B [============================================================] 0s
Writing manifest to image destination
Storing signatures
--> b05f036107d096578f7ced852104d3a9506079f77cbc3c03cbcbc9b88d4d8f27

$ sudo podman inspect quay.io/miabbott/podman-label-test:single
[                  
    {
        "Id": "b05f036107d096578f7ced852104d3a9506079f77cbc3c03cbcbc9b88d4d8f27",
        "Digest": "sha256:bd815b2a4be9cf8157844cf28eb86b8757c3c533fff8a2134bcd2bb0296dafde",
        "RepoTags": [
            "quay.io/miabbott/podman-label-test:single"
        ],
        "RepoDigests": [
            "quay.io/miabbott/podman-label-test@sha256:bd815b2a4be9cf8157844cf28eb86b8757c3c533fff8a2134bcd2bb0296dafde"
        ],
        "Parent": "",
        "Comment": "Created by Image Factory",
        "Created": "2018-07-24T14:14:20.42654769Z",
        "ContainerConfig": {
            "Env": [
                "DISTTAG=f28container",
                "FGC=f28",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],
            "Entrypoint": [
                "/bin/sh",
                "-c",
                "\"/bin/bash\""
            ],
            "WorkingDir": "/",
            "Labels": {
                "io.openshift.commit": "= deadbeef",
                "io.openshift.version": "= 3.11",
                "license": "MIT",
                "name": "fedora",
                "vendor": "Fedora Project",
                "version": "28"
            }
        },
        "Version": "",
        "Author": "",
        "Architecture": "amd64",
        "Os": "linux",
        "Size": 262873291,
        "VirtualSize": 262873291,
        "GraphDriver": {
            "Name": "overlay",
            "Data": {
                "LowerDir": "/var/lib/containers/storage/overlay/4656e146fab7f95ad90808bab5aa27e0041f1fa2427597a68efc6fa66693689d/diff",
                "MergedDir": "/var/lib/containers/storage/overlay/091bcce0127a47ee3e4342cbc6fe065b2a7f0787718ce7dab7e755031df09902/merged",
                "UpperDir": "/var/lib/containers/storage/overlay/091bcce0127a47ee3e4342cbc6fe065b2a7f0787718ce7dab7e755031df09902/diff",
                "WorkDir": "/var/lib/containers/storage/overlay/091bcce0127a47ee3e4342cbc6fe065b2a7f0787718ce7dab7e755031df09902/work"
            }
        },
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:4656e146fab7f95ad90808bab5aa27e0041f1fa2427597a68efc6fa66693689d",
                "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef"
            ]
        },
        "Labels": {
            "io.openshift.commit": "= deadbeef",
            "io.openshift.version": "= 3.11",
            "license": "MIT",
            "name": "fedora",
            "vendor": "Fedora Project",
            "version": "28"
        },
        "Annotations": {},
        "ManifestType": "application/vnd.oci.image.manifest.v1+json",
        "User": ""
    }
]

COPY --from doesn't fail when source file doesn't exist

when copying file from previous stage
via COPY --from=previous_stage
build doesn't fail on COPY command when source file doesn't exist

--> COPY --from=stage1 /opt/non-existent /file_from_stage1
--> RUN cat /file_from_stage1
cat: /file_from_stage1: No such file or directory
running 'cat /file_from_stage1' failed with exit code 1

starting container with HostConfig was deprecated since v1.10 and removed in v1.12

On a machine with docker v1.12 it throws above error error, but works fine on docker v1.10.3

running:

$ imagebuilder . surajkanginx
--> Image fedora was not found, pulling ...
--> Pulled 0/1 layers, 7% complete
--> Pulled 1/1 layers, 100% complete
--> Extracting
--> FROM fedora
unable to start build container: API error (400): {"message":"starting container with HostConfig was deprecated since v1.10 and removed in v1.12"}

dockerfile:

 $ cat Dockerfile 
FROM fedora
MAINTAINER http://fedoraproject.org/wiki/Cloud

RUN dnf -y update && dnf clean all
RUN dnf -y install nginx && dnf clean all
RUN echo "daemon off;" >> /etc/nginx/nginx.conf
RUN echo "nginx on Fedora" > /usr/share/nginx/html/index.html

EXPOSE 80

CMD [ "/usr/sbin/nginx" ]

docker version

$ docker version
Client:
 Version:      1.12.0
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   8eab29e
 Built:        
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.0
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   8eab29e
 Built:        
 OS/Arch:      linux/amd64

Dockerfile reference

Hi,
I am in the situation where I have to do docker builds in docker containers. Your solution through buildah interests me and would avoid the docker sock mapping.
What dockerfile instructions are supported?
I did not find any information in the OCI specifications.

Is there a corresponding docker version from which dockerfile instructions are compatible with imagebuilder usage?

go get does not work

Unless GOPATH includes the vendored directory either from Origin or from this repo.

Imagebuilder doesn't copy .dockerignore into the image

e.Excludes = append(excludes, ".dockerignore")

This is inconsistent with docker. I'm trying to add imagebuilder as an option in our CI to validate we work with it before getting our component into OSBS, but part of our jobs validates that there's no unstaged changes after re-generating some files (to ensure commited generated files look the same in CI as what the developer produced), but this causes issues with that since the .dockerignore is missing in the image after building.

Multistage builds use runtime CMD parameters as parameters for builder images entry point

When deploying an image such as the one depicted here (https://medium.com/@chemidy/create-a-small-and-secure-angular-docker-image-based-on-nginx-93452cb08aa2) OpenShift builds the multistage docker image as it should. However, at runtime, the CMD parameters set during runtime stage are passed to the CMD of the builder stage. In the case of the image described in the link above, the "nginx -g daemon off" is used as parameters for CMD stored inside of the builder image which reads CMD [ "node" ]. This results in the container starting at runtime as:
"node nginx -g daemon off" instead of "nginx -g daemon off" as it should

imageProgressWriter doesn't propagate docker errors under certain conditions

The issue is that imageProgressWriter can be used in two ways, by calling ReadFrom or by calling Write (possibly multiple times). Both go-dockerclient and unit tests pass the writer to io.Copy which uses the ReadFrom method. However when I wrap imageProgressWriter in an io.MultiWriter the Write method is used which results in slightly different behavior. For example when the decoded JSON object contains error key, ReadFrom returns the error immediately but Write returns it on the next invocation which may never happen.

For openshift/origin#12407 I have an implementation of Write that does not use io.Pipe/goroutine but it's ugly and does a lot of unnecessary copying: https://github.com/mmilata/origin/blob/25a5c910d1afb87a0f8b6266954a72ee7eca4ec8/pkg/build/builder/dockerutil.go#L253

To reproduce:

diff --git a/imageprogress/progress_test.go b/imageprogress/progress_test.go
index 2438d80..d959f5f 100644
--- a/imageprogress/progress_test.go
+++ b/imageprogress/progress_test.go
@@ -79,7 +79,7 @@ func TestReports(t *testing.T) {
                        },
                )
                w.(*imageProgressWriter).stableThreshhold = 0
-               _, err := io.Copy(w, pipeIn)
+               _, err := io.Copy(io.MultiWriter(w), pipeIn)
                if err != nil {
                        if !test.errExpected {
                                t.Errorf("%s: unexpected: %v", test.name, err)
@@ -115,7 +115,7 @@ func TestErrorOnCopy(t *testing.T) {
        w := newWriter(func(r report) {}, func(a, b report) bool { return true })
 
        // Ensure that the error is propagated to the copy
-       _, err := io.Copy(w, genIn)
+       _, err := io.Copy(io.MultiWriter(w), genIn)
        if err == nil {
                t.Errorf("Did not get an error when copying to writer")
        }

godep on imagebuilder causes problems with old packages

I have written this script, following README. But with this I get issues while compiling, about fsouza/go-dockerclient used from different locations when doing godep restore and godep save it causes some conflicts with the dependencies used in imagebuilder, as github.com/docker/docker/builder/command not found, it seems to have been removed from upstream docker.

    docker "github.com/fsouza/go-dockerclient"
...
    "github.com/openshift/imagebuilder/dockerclient"
    deployapi "github.com/openshift/origin/pkg/deploy/api"
)

func buildImagesRemote(buildDirPath string, imageName string) error {
    f, err := os.Open(filepath.Join(buildDirPath, "Dockerfile"))
    if err != nil {
        return err
    }
    defer f.Close()

    client, err := docker.NewClientFromEnv()
    if err != nil {
        logrus.Fatal(err)
    }
    e := dockerclient.NewClientExecutor(client)
    e.Out, e.ErrOut = os.Stdout, os.Stderr
    e.AllowPull = true
    e.Directory = buildDirPath
    e.Tag = imageName
    e.AuthFn = nil // ... pass a function to retrieve authorization info
    e.LogFn = func(format string, args ...interface{}) {
        fmt.Fprintf(e.ErrOut, "--> %s\n", fmt.Sprintf(format, args...))
    }

    buildErr := e.Build(f, map[string]string)
    if err := e.Cleanup(); err != nil {
        fmt.Fprintf(e.ErrOut, "error: Unable to clean up build: %v\n", err)
    }
    return nil
}

But when compiling I get errors as:

+ go build -tags experimental -o kompose ./cli/main/
# github.com/skippbox/kompose/pkg/transformer/kubernetes
pkg/transformer/kubernetes/k8sutils.go:55: cannot use client (type
*"github.com/skippbox/kompose/vendor/github.com/fsouza/go-dockerclient".Client) as type
*"github.com/openshift/imagebuilder/vendor/github.com/fsouza/go-dockerclient".Client in
argument to dockerclient.NewClientExecutor
pkg/transformer/kubernetes/k8sutils.go:65: type map[string]string is not an expression

Tried doing godep restore and godep save ./... but then facing this issue of

$ godep save ./...
godep: Package (github.com/docker/docker/builder/command) not found

don't abuse stderr

imagebuilder outputs Dockerfile operations to stderr:

$ git clone https://github.com/sosiouxme/scratch -b 20180626-overwrite-dir
$ cd scratch
[scratch] $ imagebuilder . > /dev/null
--> FROM fedora:28 as 0
--> ADD some-binary /usr/bin
--> RUN echo
running 'echo' failed with exit code 126

This didn't seem like much of a problem at first, however when running this from OSBS it means the output is being read from two streams and the output can get scrambled if one buffers while the other flows. Also if there is an error, it would be good to know what part of the output was actually error (right now I just use the last line of stderr which seems OK but I don't like that assumption).

VOLUME contains data added after declaration

I understand this is a known issue, that a Dockerfile like the following will end up creating a volume differently when built using imagebuilder:

FROM busybox
RUN mkdir /vol
RUN touch /vol/foo
VOLUME /vol
RUN touch /vol/bar

It may be that a volume is carefully set up in a Dockerfile, then some other command is run which -- as a side effect -- modifies that volume content. The author may be using the knowledge that the volume declaration 'freezes' the initial volume content in order to know the side-effect-producing command is safe to run.

I'd like to propose a solution: during the implementation of VOLUME, imagebuilder could export the content of the container and capture the content. This is the author's intended initial content for the volume.

After all the Dockerfile instructions are run, a final set of instructions could restore the content of volumes declared in the Dockerfile to the state they were in at declaration. For instance, saving the current user, adding a binary to clean out volume content (and afterwards, itself), becoming root and running it, then restoring the current user.

binary builds

Hello, can there be binary builds so one doesn't have to compile on every machine? CentOS and Fedora for example.

ADD doesn't expand archives

There's this fun Docker behavior where ADD (but not COPY!) will expand a tarball instead of just copying it into place as-is.

If <src> is a local tar archive in a recognized compression format (identity, gzip, bzip2 or xz) then it is unpacked as a directory. Resources from remote URLs are not decompressed. ... Whether a file is identified as a recognized compression format or not is done solely based on the contents of the file, not the name of the file.

imagebuilder doesn't do that. This would be useful to expand source archives from lookaside in OSBS. You can work around it with RUN tar fx or whatever but it's one more difference to have to account for.

๐Ÿš cd $(mktemp -d)
[tmp.aqbI4PmaDS] ๐Ÿš touch foo 
[tmp.aqbI4PmaDS] ๐Ÿš tar fc foo.tar foo
[tmp.aqbI4PmaDS] ๐Ÿš cat <<DF > Dockerfile
> FROM fedora:28
> WORKDIR /tmp
> ADD foo.tar .
> RUN ls | grep foo
> DF
[tmp.aqbI4PmaDS] ๐Ÿš imagebuilder .
--> FROM fedora:28 as 0
--> WORKDIR /tmp
--> ADD foo.tar .
--> RUN ls | grep foo
foo.tar
--> Committing changes ...
--> Done
[tmp.aqbI4PmaDS] ๐Ÿš 
[tmp.aqbI4PmaDS] ๐Ÿš 
[tmp.aqbI4PmaDS] ๐Ÿš docker build .
Sending build context to Docker daemon 13.31 kB
Step 1/4 : FROM fedora:28
 ---> 75aeb7f897fd
Step 2/4 : WORKDIR /tmp
 ---> Using cache
 ---> 41c3743a8378
Step 3/4 : ADD foo.tar .
 ---> 329f459cf92a
Removing intermediate container 80b097ebbd16
Step 4/4 : RUN ls | grep foo
 ---> Running in 34e030016475
foo
 ---> c5ae75b6e86b
Removing intermediate container 34e030016475
Successfully built c5ae75b6e86b

Only send needed files as Docker context (use Buildkit)

Today I searched for a method to enhance the performance for a Docker Build in OpenShift by only sending those files as Docker context, which we actually copy/add/use in the Dockerfile.
Using a .dockerignore file was no solution, since we use multiple repositories and I would have needed multiple compound ignore files.

Then I found out about Buildkit and how it only sends those files which are used in the Dockerfile - exactly what I wanted.

None of the OpenShift docs mention Buildkit (search results).
Then I searched in the openshift organsization on GitHub, but no relevant Code results came up.
The only relevant result was #90, which was closed.
Originally posted by @rhatdan in #90 (comment)

Since this did not actually fix the issue, and the issue I saw failed the same way on Docker, I am closing this.

I found a commit similar to the preceding PR 67132c9, which does not seem to be in either of the current branches.

How can I use Buildkit for Docker BuildConfigs?

"COPY /blah/blub /path" does not work same way as docker

The COPY command does not work for the following case - source files are not interpreted as relativ to the current workspace as stated in the original docker refence:

https://docs.docker.com/engine/reference/#copy

Multiple <src> resources may be specified but the paths of files and directories will be interpreted as relative to the source of the context of the build.

Example workspace directory:

Dockerfile
firstdir/seconddir/dir-a/file-a
firstdir/seconddir/dir-b/file-b

Dockerfile content:

FROM alpine:3.10
COPY /firstdir/seconddir /var
RUN ls -la /var && ls -la /var/firstdir
RUN ls -la /var/dir-a

The original docker builder copies the content of the firstdir/seconddir to the image, because all paths are relative to the context. imagebuilder failes with an error message saying stat seconddir: no such file or directory

Changing the COPY line to the following (removing first slash)

COPY firstdir/seconddir /var

Imagebuilder works as expected.

generated image is broken

Hey all,
I tried building https://github.com/CentOS/sig-cloud-instance-images/blob/f5b919346432acc728078aa32ffb6dcf84d303a0/docker/Dockerfile using imagebuilder. The resulting image is broken. After a short chat with @jperrin he pointed out that there was moby/moby#20331 which could cause this behavior.

I was using docker-1.10.3-26.git1ecb834.fc24.x86_64

Incorrect ENV substitution

Use Case 1 - Label from ENV Values - Works

$ cat Dockerfile 
FROM fedora

ENV FOO="foo"

LABEL test="$FOO"

$ oc ex dockerbuild . baseimage-imagebuilder:latest
--> FROM fedora
--> ENV FOO="foo"
--> LABEL test="$FOO"
--> Committing changes to baseimage-imagebuilder:latest ...
--> Done

$ docker inspect baseimage-imagebuilder:latest | jq '.[0].Config | .Env, .Labels'
[
  "FOO=foo",
  "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
]
{
  "test": "foo"
}

Use Case 2 - Inheriting ENV Values - Works

$ cd baseimage

$ cat Dockerfile 
FROM fedora

ENV FOO="foo"

$ oc ex dockerbuild . baseimage-imagebuilder:latest
--> FROM fedora
--> ENV FOO="foo"
--> Committing changes to baseimage-imagebuilder:latest ...
--> Done

$ cd ../layeredimage/

$ oc ex dockerbuild . layered-dockerbuild:latest
--> FROM baseimage-docker-api:latest
--> LABEL test="$FOO"
--> Committing changes to layered-dockerbuild:latest ...
--> Done

$ docker inspect layered-dockerbuild:latest | jq '.[0].Config | .Env, .Labels'
[
  "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
  "FOO=foo"
]
{
  "test": "foo"
}

Use Case 3 - Overriding ENV Values - FAIL

$ cat Dockerfile 
FROM fedora

ENV FOO="foo"

$ oc ex dockerbuild . baseimage-imagebuilder:latest
--> FROM fedora
--> ENV FOO="foo"
--> Committing changes to baseimage-imagebuilder:latest ...
--> Done

$ cd ../layeredimage/

$ cat Dockerfile 
FROM baseimage-docker-api:latest

ENV FOO="oof"
LABEL test="$FOO"

$ oc ex dockerbuild . layered-dockerbuild:latest
--> FROM baseimage-docker-api:latest
--> ENV FOO="oof"
--> LABEL test="$FOO"
--> Committing changes to layered-dockerbuild:latest ...
--> Done

$ docker inspect layered-dockerbuild:latest | jq '.[0].Config | .Env, .Labels'
[
  "FOO=oof",
  "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
]
{
  "test": "foo"
}

Notice how, even though the Dockerfile for the layered image overrides the FOO environment variable, it's not applied to the test label.

build failure with recent version of docker/docker

Vendoring recent version of docker/docker causes build failures:

vendor/github.com/openshift/imagebuilder/builder.go:16:2: cannot find package "github.com/docker/docker/builder/dockerfile/command" in any of:

It seems that the dockerfile package has been refactored into a single package (no sub-packages anymore).

ProcessWord{,s} removes quotes

When using buildah, the content of specified labels will be altered in a way that specified quotes get removed. This can lead to various kinds of bugs and undesired side-effects.

If, for instance, a Dockerfile specifies LABEL RUN /usr/bin/foo --string-arg "a string arg" removing the quotes is likely to cause issues when the image is being used in special contexts (e.g., podman container runlabel).

This issue affects Podman and Buildah. Note that Docker has the very same issue, which I consider to be a bug for the reasons mentioned above. I had a look at the code and it seems fixable but I am not overly familiar with it and want to collect some feedback before.

Cc @baude @mheon @TomSweeneyRedHat @rhatdan

@baude, we could reuse the functions for runlabel's env replacements.

failed with exit code 3

This Dockerfile apparently caused a bug:

$ git clone https://github.com/openshift/openshift-ansible.git
$ cd openshift-ansible
$ git checkout 18961e08a744981f0a7726f653edf0c981ff67e6
$ imagebuilder -f images/installer/Dockerfile -t ib_test .

[... a bunch of yum stuff...]
Complete!
package python2-passlib is not installed
package origin-clients is not installed
package google-cloud-sdk-183.0.0 is not installed
running '/bin/sh -c export "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"; INSTALL_PKGS="python-lxml python-dns pyOpenSSL python2-cryptography openssl java-1.8.0-openjdk-headless python2-passlib httpd-tools openssh-clients origin-clients"  && yum install -y --setopt=tsflags=nodocs $INSTALL_PKGS  && EPEL_PKGS="python2-boto python2-boto3 python2-crypto google-cloud-sdk-183.0.0 which python2-pip.noarch"  && EPEL_TESTING_PKGS="ansible"  && yum install -y epel-release  && yum install -y --setopt=tsflags=nodocs $EPEL_PKGS  && yum install -y --setopt=tsflags=nodocs --enablerepo=epel-testing $EPEL_TESTING_PKGS  && rpm -V $INSTALL_PKGS $EPEL_PKGS $EPEL_TESTING_PKGS  && pip install apache-libcloud~=2.2.1  && yum clean all' failed with exit code 3

docker build worked fine.

(There's a bunch of yum stuff up front that can probably be commented out to reproduce faster -- dunno, I'm just dashing this off)

directory not created with WORKDIR and VOLUME

$ cat Dockerfile

FROM centos:7
WORKDIR /foo
VOLUME  [ "/foo" ]
RUN echo

$ imagebuilder .
--> FROM centos:7 as 0
--> WORKDIR /foo
--> VOLUME  [ "/foo" ]
--> RUN echo
API error (404): {"message":"lstat /var/lib/docker/overlay2/6b7c9551518e64992927f58da362b38949eddbb602f5eea6e15a9278be278bdb/merged/foo/: no such file or directory"}

You will also get a different error without the VOLUME line:

/bin/sh: line 0: cd: /foo: No such file or directory

Docker creates the dir implicitly.

The workaround is to RUN mkdir /foo before using it as a volume or workdir.

Builds on remote machine?

Can I invoke builds on remote machine that has docker and openshift running?

So my use case is I am developer and for developing apps that run on OpenShift I use CDK and I don't have docker daemon on my host machine, so I would like to do image builds on CDK machine and use those built images on the openshift running on CDK itself?

So I would like to use this library in tool like kompose where a user can write docker-compose file and run kompose on the host machine which converts the docker-compose to openshift artifacts and deploy it in CDK, where all these builds also happen?

exec: "sleep 86400": executable file not found in $PATH

I must be doing something wrong here but can't figure out what.

With imagebuilder from latest head, and docker-1.13.1-51.git4032bd5.fc27.x86_64

$ imagebuilder -t foo -f Dockerfile.min .
--> FROM fedora:27 as 0
--> RUN touch /etc/foo
--> Committing changes to foo ...
--> Done
$ docker run -it --rm foo 
/usr/bin/docker-current: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: \"sleep 86400\": executable file not found in $PATH".
$ docker run -it --rm foo /bin/bash
[root@7e2cdeabd622 /]# exit
$ docker run -it --rm foo sleep 1
$

First of all, where did "sleep 86400" come from? fedora:27 has no Cmd or Entrypoint set. foo does.

$ docker run -it --rm fedora:27
/usr/bin/docker-current: Error response from daemon: No command specified.
See '/usr/bin/docker-current run --help'.
$ docker inspect foo
[
    {
        "Id": "sha256:948b2333e11c0dfbedf4bde7e7a7fe6749376dd5d5fae112992edfa600838ad8",
        "RepoTags": [
            "foo:latest"
        ],
        "RepoDigests": [],
        "Parent": "sha256:9110ae7f579f35ee0c3938696f23fe0f5fbe641738ea52eb83c2df7e9995fa17",
        "Comment": "",
        "Created": "2018-04-18T19:01:37.421678496Z",
        "Container": "fc1eb310d84b08683a2eab434268e46313bdf46b2a533606e4cdbed1a5e7727e",
        "ContainerConfig": {
            "Hostname": "fc1eb310d84b",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "DISTTAG=f27container",
                "FGC=f27",
                "FBR=f27"
            ],
            "Cmd": [
                "sleep 86400"
            ],
            "Image": "fedora:27",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": [
                "/bin/sh",
                "-c"
            ],
            "OnBuild": null,
            "Labels": {}
        },
        "DockerVersion": "1.13.1",
        "Author": "",
        "Config": {
            "Hostname": "",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "DISTTAG=f27container",
                "FGC=f27",
                "FBR=f27"
            ],
            "Cmd": [
                "sleep 86400"
            ],
            "Image": "",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": [],
            "OnBuild": null,
            "Labels": {}
        },
        "Architecture": "amd64",
        "Os": "linux",
        "Size": 235250693,
        "VirtualSize": 235250693,
        "GraphDriver": {
            "Name": "overlay2",
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/70bea816b920919e3afcdaca1f7c572c83308f4bee6379450e17acba57da28ff/diff",
                "MergedDir": "/var/lib/docker/overlay2/6e1b01b002e68b2410fe67bcfee1bc7647346563074c82bc1abc534681459b01/merged",
                "UpperDir": "/var/lib/docker/overlay2/6e1b01b002e68b2410fe67bcfee1bc7647346563074c82bc1abc534681459b01/diff",
                "WorkDir": "/var/lib/docker/overlay2/6e1b01b002e68b2410fe67bcfee1bc7647346563074c82bc1abc534681459b01/work"
            }
        },
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:39bae602f7539e626f6894ecded6d12a6c56745dbab9aee940e258c766e090e8",
                "sha256:de41557ebc416b99c4bb90ae5e39f4c451055236db8aff0cbe10f8206a275039"
            ]
        }
    }
]

Second, why does it fail to run that?

`go get` fails to compile

With golang-1.7.3-0.2.el7.x86_64:

failed: [172.18.13.178] (item=github.com/openshift/imagebuilder/cmd/imagebuilder) => {
    "changed": true, 
    "cmd": [
        "go", 
        "get", 
        "github.com/openshift/imagebuilder/cmd/imagebuilder"
    ], 
    "delta": "0:00:33.606174", 
    "end": "2017-04-13 14:49:42.627503", 
    "failed": true, 
    "item": "github.com/openshift/imagebuilder/cmd/imagebuilder", 
    "rc": 2, 
    "start": "2017-04-13 14:49:09.021329", 
    "stderr": [
        "# github.com/openshift/imagebuilder", 
        "/data/src/github.com/openshift/imagebuilder/evaluator.go:15: undefined: parser.SetEscapeToken", 
        "/data/src/github.com/openshift/imagebuilder/evaluator.go:18: too many arguments in call to parser.Parse", 
        "/data/src/github.com/openshift/imagebuilder/evaluator.go:18: cannot use *parser.Result as type *parser.Node in return argument"
    ], 
    "stdout": [], 
    "warnings": []
}

here

/cc @smarterclayton

ADD foo /usr/bin nukes /usr/bin

$ git clone https://github.com/sosiouxme/scratch -b 20180626-overwrite-dir
$ cd scratch


[scratch] $ docker build .
Sending build context to Docker daemon 61.44 kB
Step 1/3 : FROM fedora:28
 ---> 75aeb7f897fd
Step 2/3 : ADD some-binary /usr/bin
 ---> ad0d2f7819ca
Removing intermediate container 777b88ad62b1
Step 3/3 : RUN echo
 ---> Running in e9bd7b6ca65a

 ---> 511014a1d278
Removing intermediate container e9bd7b6ca65a
Successfully built 511014a1d278


[scratch] $ imagebuilder .
--> FROM fedora:28 as 0
--> ADD some-binary /usr/bin
--> RUN echo
rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused "exec: \"/bin/sh\": stat /bin/sh: not a directory"

running 'echo' failed with exit code 126

Rev of glog is not checkout'able

looking at the Godeps it turns out that the revision of glog used cant be checked out:

$ git clone https://github.com/golang/glog
$ cd glog/
$ git checkout 335da9dda11408a34b64344f82e9c03779b71673
fatal: reference is not a tree: 335da9dda11408a34b64344f82e9c03779b71673

... or I am to newbish to Go

FTBFS on Rawhide

Building commit c3e2e96:

From https://copr-be.cloud.fedoraproject.org/results/vrutkovs/new-packages/fedora-rawhide-x86_64/00570989-imagebuilder/build.log.gz:

# _/builddir/build/BUILD/imagebuilder-c3e2e96f351aa1b355fb90169ec9390b7eff5fc5
./dispatchers.go:386: b.RunConfig.StopSignal undefined (type docker.Config has no field or method StopSignal)
./dispatchers.go:386: b.RunConfig.StopSignal undefined (type docker.Config has no field or method StopSignal)
./dispatchers.go:406: b.RunConfig.Healthcheck undefined (type docker.Config has no field or method Healthcheck)
./dispatchers.go:406: b.RunConfig.Healthcheck undefined (type docker.Config has no field or method Healthcheck)
./dispatchers.go:406: undefined: docker.HealthConfig
./dispatchers.go:406: undefined: docker.HealthConfig
./dispatchers.go:410: b.RunConfig.Healthcheck undefined (type docker.Config has no field or method Healthcheck)
./dispatchers.go:410: b.RunConfig.Healthcheck undefined (type docker.Config has no field or method Healthcheck)
./dispatchers.go:411: b.RunConfig.Healthcheck undefined (type docker.Config has no field or method Healthcheck)
./dispatchers.go:411: b.RunConfig.Healthcheck undefined (type docker.Config has no field or method Healthcheck)
./dispatchers.go:417: undefined: docker.HealthConfig
./dispatchers.go:417: undefined: docker.HealthConfig
./dispatchers.go:468: b.RunConfig.Healthcheck undefined (type docker.Config has no field or method Healthcheck)
./dispatchers.go:468: b.RunConfig.Healthcheck undefined (type docker.Config has no field or method Healthcheck)

Spec: http://copr-dist-git.fedorainfracloud.org/cgit/vrutkovs/new-packages/imagebuilder.git/tree/imagebuilder.spec?id=023ddfd4681b876c997be8351ede76356108ef0b

Full build log: https://copr-be.cloud.fedoraproject.org/results/vrutkovs/new-packages/fedora-rawhide-x86_64/00570989-imagebuilder/build.log.gz

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.