deis / dockerbuilder Goto Github PK

Deis dockerbuilder builds your app from a Dockerfile inside a Kubernetes pod, used by Deis Workflow.

License: MIT License

Makefile 23.42% Python 76.58%

deis-workflow kubernetes k8s dockerfile python3 unsupported

dockerbuilder's Introduction

	This repository (`deis/deis`) is no longer developed or maintained. The Deis v1 PaaS based on CoreOS Container Linux and Fleet has been replaced by Deis Workflow which is based on Kubernetes.

Deis v1 PaaS

Deis (pronounced DAY-iss) is an open source PaaS that makes it easy to deploy and manage applications on your own servers. Deis builds upon Docker and CoreOS to provide a lightweight PaaS with a Heroku-inspired workflow.

New to Deis? Learn more about Deis Concepts, Architecture and how to Deploy an Application.

Installing Deis v1

Deis is a set of Docker containers that can be deployed anywhere including public cloud, private cloud, bare metal or your workstation. Decide where you'd like to deploy Deis, then follow the provider-specific documentation for provisioning.

Trying out Deis? Please follow the documentation on getting set up with Vagrant. Upgrading from a previous Deis release? See Upgrading Deis for additional information.

Troubleshooting

See the Troubleshooting Deis documentation for assistance with common issues.

Contributing

Interested in contributing to Deis? Check out our Open Roadmap and Planning Process or jump right into hacking on Deis and testing your Deis cluster.

License

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

dockerbuilder's People

Contributors

Stargazers

Watchers

dockerbuilder's Issues

Add information to the README on how to configure dockerbuilder for standalone use

cc/ @smothiki @kmala

Docker-py doesn't support insecure registries anymore

docker-py==1.7.0 doesn't support insecure registries.
There's a deprecation message when client.push() is called with insecure_registry=True:
https://github.com/docker/docker-py/blob/1.7.0/docker/api/image.py#L209

It tries to connect via HTTPS so https://github.com/deis/helloworld can't be built:

pushing to registry
remote: ":"The push refers to a repository [10.254.211.95:80/deis-helloworld](len: 1)"}

{"errorDetail":{"message":"unable to ping registry endpoint https://10.254.211.95:80/v0/\nv2 ping attempt failed with error: Get https://10.254.211.95:80/v2/: tls: oversized record received with length 20527\n v1 ping attempt failed with error: Get https://10.254.211.95:80/v1/_ping: tls: oversized record received with length 20527"},"error":"unable to ping registry endpoint https://10.254.211.95:80/v0/\nv2 ping attempt failed with error: Get https://10.254remote: 80/v2/: tls: oversized record received with length 20527\n v1 ping attempt failed with error: Get https://10.254.211.95:80/v1/_ping: tls: oversized record received with length 20527"}

Upgrade to the new object-storage-cli version

Since deis/object-storage-cli#12 is done, this component should use the new version to suppress debug output when DEIS_DEBUG is set

Support for pulling a private FROM

When workflow is configured to use a private registry the dockerbuilder doesn't provide any authentication to the docker client when building so private FROM images can't be pulled.

Expected behaviour is that shared base images are pulled using the authentication details configured for the off-cluster registry.

Actual behaviour:

No authentication details are available to the client used to build:

Starting build... but first, coffee!
Step 1 : FROM quay.io/clearbit/ruby:2.1.10
Error: Status 403 trying to pull repository clearbit/ruby: "{\"error\": \"Permission Denied\"}"
remote: 2016/09/29 09:49:09 Error running git receive hook [Build pod exited with code 1, stopping build.]

One possible solution would be to share a docker.Client() between the build and push and call client.login prior to building, rather than passing auth details through to the push specifically in https://github.com/deis/dockerbuilder/blob/master/rootfs/deploy.py#L69

Error pushing from dockerbuilder to registry

Hi everyone!

SO: CoreOS
Deis: 2.1.0
Kubernetes: 1.2.4
Testing App: helloworld example from Deis

I'm facing an issue with Deis 2.1.0 and Dockerbuilds. The message I get from the dockerbuild pod when executing the example helloworld is:

Successfully built b61d0cb0c964
Pushing to registry
    data = reader.read(1)
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/response.py", line 320, in read
    flush_decoder = True
  File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/response.py", line 233, in _error_catcher
    raise ReadTimeoutError(self._pool, None, 'Read timed out.')
requests.packages.urllib3.exceptions.ReadTimeoutError: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out.

I've seen this same message in Deis and Docker issues, but none of the solutions I've read have worked so far.
#60

Docker daemon includes: ... --insecure-registry=10.0.0.0/8 --selinux-enabled=false ...

The first thing that doesn't seem right in my nodes is that there are a couple of envs for the registry service:

DEIS_REGISTRY_SERVICE_HOST=10.200.162.204
DEIS_REGISTRY_SERVICE_PORT=80

and deploy.py should be using those

registry = os.getenv("DEIS_REGISTRY_SERVICE_HOST") + ":" + os.getenv("DEIS_REGISTRY_SERVICE_PORT")

but the error says (host='localhost', port=None): Read timed out.

remove the vendor directory

it looks like it was copied over from the prototype repo

Add storage region to dockerfile POD

amazon S3 url endpoint requires s3 region

Support of cache_from attribute

Would be nice to be able to use cache_from attribute somehow to do more some cache resolution.

Cf. https://docker-py.readthedocs.io/en/stable/images.html

dockerbuilder should delete the image from the host once the push is done

reduces storage on the host

Docker python library broken?

Hi Deis Devs,

We are encountering an error with the registry-proxy component as it calls the deisbuilder component when we try to pull the example docker application. The reason for the error may be that the Docker python library is picking the wrong api version (v1 vs. v2) when it hits the dockerbuilder with the POST.

INFO [global-inventor]: build global-inventor-d58d621 created
INFO:api.models.app:[global-inventor]: build global-inventor-d58d621 created
INFO [global-inventor]: admin deployed deis/example-go
INFO:api.models.app:[global-inventor]: admin deployed deis/example-go
INFO Pulling Docker image deis/example-go:latest
INFO:registry.dockerclient:Pulling Docker image deis/example-go:latest
INFO Tagging Docker image deis/example-go:latest as localhost:5555/global-inventor:v8
INFO:registry.dockerclient:Tagging Docker image deis/example-go:latest as localhost:5555/global-inventor:v8
INFO Pushing Docker image localhost:5555/global-inventor:v8
INFO:registry.dockerclient:Pushing Docker image localhost:5555/global-inventor:v8
ERROR:backoff:Backing off push(...) for 0.9s (registry.dockerclient.RegistryException: Put http://localhost:5555/v1/repositories/global-inventor/: dial tcp [::1]:5555: getsockopt: connection refused)
INFO Pushing Docker image localhost:5555/global-inventor:v8
INFO:registry.dockerclient:Pushing Docker image localhost:5555/global-inventor:v8
ERROR:backoff:Backing off push(...) for 1.7s (registry.dockerclient.RegistryException: Put http://localhost:5555/v1/repositories/global-inventor/: dial tcp [::1]:5555: getsockopt: connection refused)
INFO Pushing Docker image localhost:5555/global-inventor:v8
INFO:registry.dockerclient:Pushing Docker image localhost:5555/global-inventor:v8
ERROR:backoff:Giving up push(...) after 3 tries (registry.dockerclient.RegistryException: Put http://localhost:5555/v1/repositories/global-inventor/: dial tcp [::1]:5555: getsockopt: connection refused)
INFO [global-inventor]: Put http://localhost:5555/v1/repositories/global-inventor/: dial tcp [::1]:5555: getsockopt: connection refused
INFO:api.models.app:[global-inventor]: Put http://localhost:5555/v1/repositories/global-inventor/: dial tcp [::1]:5555: getsockopt: connection refused
ERROR:root:Put http://localhost:5555/v1/repositories/global-inventor/: dial tcp [::1]:5555: getsockopt: connection refused
Traceback (most recent call last):
  File "/app/api/models/release.py", line 89, in new
    release.publish()
  File "/app/api/models/release.py", line 136, in publish
    publish_release(source_image, self.image, deis_registry, self.get_registry_auth())
  File "/app/registry/dockerclient.py", line 195, in publish_release
    return DockerClient().publish_release(source, target, deis_registry, creds)
  File "/app/registry/dockerclient.py", line 114, in publish_release
    self.push("{}/{}".format(self.registry, name), tag)
  File "/usr/local/lib/python3.5/dist-packages/backoff/_sync.py", line 85, in retry
    ret = target(*args, **kwargs)
  File "/app/registry/dockerclient.py", line 131, in push
    log_output(stream, 'push', repo, tag)
  File "/app/registry/dockerclient.py", line 174, in log_output
    stream_error(chunk, operation, repo, tag)
  File "/app/registry/dockerclient.py", line 191, in stream_error
    raise RegistryException(message)
registry.dockerclient.RegistryException: Put http://localhost:5555/v1/repositories/global-inventor/: dial tcp [::1]:5555: getsockopt: connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/app/api/models/build.py", line 63, in create
    source_version=self.version
  File "/app/api/models/release.py", line 96, in new
    raise DeisException(str(e)) from e
api.exceptions.DeisException: Put http://localhost:5555/v1/repositories/global-inventor/: dial tcp [::1]:5555: getsockopt: connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/rest_framework/views.py", line 480, in dispatch
    response = handler(request, *args, **kwargs)
  File "/app/api/views.py", line 185, in create
    return super(AppResourceViewSet, self).create(request, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/rest_framework/mixins.py", line 21, in create
    self.perform_create(serializer)
  File "/app/api/viewsets.py", line 21, in perform_create
    self.post_save(obj)
  File "/app/api/views.py", line 268, in post_save
    self.release = build.create(self.request.user)
  File "/app/api/models/build.py", line 79, in create
    raise DeisException(str(e)) from e
api.exceptions.DeisException: Put http://localhost:5555/v1/repositories/global-inventor/: dial tcp [::1]:5555: getsockopt: connection refused
10.42.0.8 "POST /v2/apps/global-inventor/builds/ HTTP/1.1" 400 124 "Deis Client v2.13.0"

Notice the 10.42.0.8 "POST /v2/apps/global-inventor/builds/ HTTP/1.1" 400 124 "Deis Client v2.13.0" at the end of the log.

Is this related to deis/registry#64 (comment) and how do we get around it?

use Go SDK to download from minio

use https://github.com/minio/minio-go instead of using the mc binary. obsoletes #2

Investigate removing musl deps in Dockerfile

be able to talk to a remote docker daemon

Errors if unicode in Job output

When the Dockerfile produces Unicode output (i believe npm install prints a check sign or something) I get this error:

Traceback (most recent call last):
  File "/deploy.py", line 106, in <module>
    log_output(stream, True)
  File "/deploy.py", line 25, in log_output
    print(stream_chunk.replace('\n', ''))
UnicodeEncodeError: 'ascii' codec can't encode characters in position 264-266: ordinal not in range(128)
size of streamed logs 2173
Waiting for the deis/dockerbuild-valued-quadrant-febcad02-8068a452 pod to end. Checking every 100ms for 15m0s
Done
remote: 2016/04/19 17:09:05 Error running git receive hook [Build pod exited with code 1, stopping build.]
Checking for builder pod exit code
To ssh://[email protected]:2222/valued-quadrant.git
   d9c7b3a..febcad0  1.3 -> 1.3

I believe it is this line:

dockerbuilder/rootfs/deploy.py

Line 25 in 6066296

print(stream_chunk.replace('\n', ''))

DEBUG is 1

Publish immutable artifact tags + canary version to Docker registries

We want to have the ability to have deis/charts reference a specific immutable image which will enable us better traceability on issues that come into various components.

What we've done with efforts in other repositories is publish a version:

git-[short sha] that will never be overwritten
canary that will move to reference the latest version available of a component

Examples of the easiest way to roll this change out can be seen in deis/workflow#487, deis/builder#246, and deis/workflow-manager#11

add to CI

just as other repos do, travis.ci needs to build this codebase. since the build needs to push to dockerhub and quay.io, this issue implicitly will require a makefile refactor. see https://github.com/deis/minio/pull/9/files#r46733280 for more

update the README

It should have information specific to this component. See https://github.com/deis/workflow, https://github.com/deis/builder, or https://github.com/deis/router for inspiration

Improve testing for dockerbuilder

As of now dockerbuilder is a part of full E2E test. But if we look at functionality here dockerbuilder only does two things in DEIS. Takes a tar file extracts the code, builds a docker container according to the Dockerfile and pushes the container to registry.
Any PR in dockerbuilder should or also do this test instead of running entire E2E suite.

refer deis/workflow-e2e#303

manifest for deis dockerbuilder pod should be called 'deis-dockerbuilder-pod.yaml'

it's currently called deis-dockerbuilder.yaml

Error pushing Dockerfile with workflow-beta4

Pushing a simple Dockerfile, I'm getting the error below.

Pushing to registry {"errorDetail":{"message":"unable to ping registry endpoint https://10.0.250.51:80/v0/\nv2 ping attempt failed with error: Get https://10.0.250.51:80/v2/: tls: oversized record received with length 20527\n v1 ping attempt failed with error: Get https://10.0.250.51:80/v1/_ping: tls: oversized record received with length 20527"},"error":"unable to ping registry endpoint https://10.0.250.51:80/v0/\nv2 ping attempt failed with error: Get https://10.0.250.51:80/v2/: tls: oversized record received with length 20527\n v1 pingremote: failed with error: Get https://10.0.250.51:80/v1/_ping: tls: oversized record received with length 20527"} remote: 2016/05/15 05:51:55 Error running git receive hook [Build pod exited with code 1, stopping build.]

My env variables seem reasonable.

declare -x DEIS_REGISTRY_SERVICE_HOST="10.0.250.51" declare -x DEIS_REGISTRY_SERVICE_PORT="80" declare -x DEIS_REGISTRY_SERVICE_PORT_HTTP="80"

Is docker-py incorrectly prepending https:// ?

print("Pushing to registry") 118 stream = client.push(registry+'/'+imageName, tag=imageTag, stream=True, insecure_registry=True)

git push of Dockerfile fails

During v2.1.0 testing, I have discovered that while using off-cluster object storage (I'm using S3), the following lines throw an error, which causes the dockerbuilder pod to fail, in turn, the git receive hook on the builder fails.

https://github.com/deis/dockerbuilder/blob/master/rootfs/deploy.py#L42-L43

Logs from the failed git push:

$ git push deis master
Warning: Permanently added '[deis-builder.krancour2.deis.ninja]:2222,[54.191.146.37]:2222' (RSA) to the list of known hosts.
Counting objects: 78, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (63/63), done.
Writing objects: 100% (78/78), 12.61 KiB | 0 bytes/s, done.
Total 78 (delta 20), reused 27 (delta 4)
Starting build... but first, coffee!
remote: 2016/06/30 02:15:37 Error running git receive hook [watching events for builder pod startup (Giving up; pod went into failed status: 
remote: &api.Pod{TypeMeta:unversioned.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:api.ObjectMeta{Name:"dockerbuild-metric-duckling-5d313d57-24d031d0", GenerateName:"", Namespace:"deis", SelfLink:"/api/v1/namespaces/deis/pods/dockerbuild-metric-duckling-5d313d57-24d031d0", UID:"875da86a-3e68-11e6-a019-0674990da1bb", ResourceVersion:"59976", Generation:0, CreationTimestamp:unversioned.Time{Time:time.Time{sec:63602849734, nsec:0, loc:(*time.Location)(0x22f6a20)}}, DeletionTimestamp:(*unversioned.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"heritage":"dockerbuild-metric-duckling-5d313d57-24d031d0"}, Annotations:map[string]string(nil)}, Spec:api.PodSpec{Volumes:[]api.Volume{api.Volume{Name:"objectstorage-keyfile", VolumeSource:api.VolumeSource{HostPath:(*api.HostPathVolumeSource)(nil), EmptyDir:(*api.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*api.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*api.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*api.GitRepoVolumeSource)(nil), Secret:(*api.SecretVolumeSource)(0xc820454470), NFS:(*api.NFSVolumeSource)(nil), ISCSI:(*api.ISCSIVolumeSource)(nil), Glusterfs:(*api.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*api.PersistentVolumeClaimVolumeSource)(nil), RBD:(*api.RBDVolumeSource)(nil), FlexVolume:(*api.FlexVolumeSource)(nil), Cinder:(*api.CinderVolumeSource)(nil), CephFS:(*api.CephFSVolumeSource)(nil), Flocker:(*api.FlockerVolumeSource)(nil), DownwardAPI:(*api.DownwardAPIVolumeSource)(nil), FC:(*api.FCVolumeSource)(nil), AzureFile:(*api.AzureFileVolumeSource)(nil), ConfigMap:(*api.ConfigMapVolumeSource)(nil)}}, api.Volume{Name:"docker-socket", VolumeSource:api.VolumeSource{HostPath:(*api.HostPathVolumeSource)(0xc8204544b0), EmptyDir:(*api.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*api.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*api.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*api.GitRepoVolumeSource)(nil), Secret:(*api.SecretVolumeSource)(nil), NFS:(*api.NFSVolumeSource)(nil), ISCSI:(*api.ISCSIVolumeSource)(nil), Glusterfs:(*api.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*api.PersistentVolumeClaimVolumeSource)(nil), RBD:(*api.RBDVolumeSource)(nil), FlexVolume:(*api.FlexVolumeSource)(nil), Cinder:(*api.CinderVolumeSource)(nil), CephFS:(*api.CephFSVolumeSource)(nil), Flocker:(*api.FlockerVolumeSource)(nil), DownwardAPI:(*api.DownwardAPIVolumeSource)(nil), FC:(*api.FCVolumeSource)(nil), AzureFile:(*api.AzureFileVolumeSource)(nil), ConfigMap:(*api.ConfigMapVolumeSource)(nil)}}, api.Volume{Name:"default-token-gb00u", VolumeSource:api.VolumeSource{HostPath:(*api.HostPathVolumeSource)(nil), EmptyDir:(*api.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*api.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*api.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*api.GitRepoVolumeSource)(nil), Secret:(*api.SecretVolumeSource)(0xc8204544e0), NFS:(*api.NFSVolumeSource)(nil), ISCSI:(*api.ISCSIVolumeSource)(nil), Glusterfs:(*api.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*api.PersistentVolumeClaimVolumeSource)(nil), RBD:(*api.RBDVolumeSource)(nil), FlexVolume:(*api.FlexVolumeSource)(nil), Cinder:(*api.CinderVolumeSource)(nil), CephFS:(*api.CephFSVolumeSource)(nil), Flocker:(*api.FlockerVolumeSource)(nil), DownwardAPI:(*api.DownwardAPIVolumeSource)(nil), FC:(*api.FCVolumeSource)(nil), AzureFile:(*api.AzureFileVolumeSource)(nil), ConfigMap:(*api.ConfigMapVolumeSource)(nil)}}}, Containers:[]api.Container{api.Container{Name:"deis-dockerbuilder", Image:"quay.io/deisci/dockerbuilder:git-4f4ebb5", Command:[]string(nil), Args:[]string(nil), WorkingDir:"", Ports:[]api.ContainerPort(nil), Env:[]api.EnvVar{api.EnvVar{Name:"TAR_PATH", Value:"home/metric-duckling:git-5d313d57/tar", ValueFrom:(*api.EnvVarSource)(nil)}, api.EnvVar{Name:"IMG_NAME", Value:"metric-duckling:git-5d313d57", ValueFrom:(*api.EnvVarSource)(nil)}, api.EnvVar{Name:"BUILDER_STORAGE", Value:"s3", ValueFrom:(*api.EnvVarSource)(nil)}}, Resources:api.ResourceRequirements{Limits:api.ResourceList(nil), Requests:api.ResourceList(nil)}, VolumeMounts:[]api.VolumeMount{api.VolumeMount{Name:"objectstorage-keyfile", ReadOnly:true, MountPath:"/var/run/secrets/deis/objectstore/creds"}, api.VolumeMount{Name:"docker-socket", ReadOnly:false, MountPath:"/var/run/docker.sock"}, api.VolumeMount{Name:"default-token-gb00u", ReadOnly:true, MountPath:"/var/run/secrets/kubernetes.io/serviceaccount"}}, LivenessProbe:(*api.Probe)(nil), ReadinessProbe:(*api.Probe)(nil), Lifecycle:(*api.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", ImagePullPolicy:"Always", SecurityContext:(*api.SecurityContext)(nil), Stdin:false, StdinOnce:false, TTY:false}}, RestartPolicy:"Never", TerminationGracePeriodSeconds:(*int64)(0xc820454530), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:"ClusterFirst", NodeSelector:map[string]string(nil), ServiceAccountName:"default", NodeName:"ip-10-0-0-175.us-west-2.compute.internal", SecurityContext:(*api.PodSecurityContext)(0xc8201835c0), ImagePullSecrets:[]api.LocalObjectReference(nil)}, Status:api.PodStatus{Phase:"Failed", Conditions:[]api.PodCondition{api.PodCondition{Type:"Ready", Status:"False", LastProbeTime:unversioned.Time{Time:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}}, LastTransitionTime:unversioned.Time{Time:time.Time{sec:63602849734, nsec:0, loc:(*time.Location)(0x22f6a20)}}, Reason:"ContainersNotReady", Message:"containers with unready status: [deis-dockerbuilder]"}}, Message:"", Reason:"", HostIP:"10.0.0.175", PodIP:"10.2.68.8", StartTime:(*unversioned.Time)(0xc820483ac0), ContainerStatuses:[]api.ContainerStatus{api.ContainerStatus{Name:"deis-dockerbuilder", State:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(0xc82022d2d0)}, LastTerminationState:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(nil)}, Ready:false, RestartCount:0, Image:"quay.io/deisci/dockerbuilder:git-4f4ebb5", ImageID:"docker://sha256:819f3bbadd2a885b89efeb76f6c52b70715e0a76195a83ab8d5996faa6f8b128", ContainerID:"docker://6fc27094e9633107fb337766bc2eff3bcfe6c220bf618a5441bbf0be8cc19315"}}}})]
To ssh://[email protected]:2222/metric-duckling.git
 ! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'ssh://[email protected]:2222/metric-duckling.git'

Logs from the failed dockerbuilder pod:

$ k logs dockerbuild-metric-duckling-5d313d57-24d031d0 --namespace=deis
2016/06/30 02:15:36 open /var/run/secrets/deis/objectstore/creds/bucket: no such file or directory
Traceback (most recent call last):
  File "/deploy.py", line 48, in <module>
    download_file(tar_path)
  File "/deploy.py", line 43, in download_file
    subprocess.check_call(command)
  File "/usr/lib/python2.7/subprocess.py", line 541, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['objstorage', '--storage-type=s3', 'download', 'home/metric-duckling:git-5d313d57/tar', 'apptar']' returned non-zero exit status 1

This is a showstopper for v2.1.0.

Needs python linting

Since #6 introduces some python code, we should flake8 source files in CI as we do in deis/workflow.

Failed git push output isn't always displayed correctly

While #74 avoids a stack trace and usually prints user-friendly output, sometimes git push of a bad Dockerfile will emit different, less useful output:

$ git push deis master
Warning: Permanently added '[deis-builder.deis.rocks]:2222,[104.196.15.240]:2222' (ECDSA) to the list of known hosts.
Starting build... but first, coffee!        
remote: 2016/07/06 16:31:09 Error running git receive hook [watching events for builder pod startup (Giving up; pod went into failed status:         
remote: &api.Pod{TypeMeta:unversioned.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:api.ObjectMeta{Name:"dockerbuild-test-270779558-19a188ee-617b208a", GenerateName:"", Namespace:"deis", SelfLink:"/api/v1/namespaces/deis/pods/dockerbuild-test-270779558-19a188ee-617b208a", UID:"0a4c54d2-4397-11e6-ac90-42010af00026", ResourceVersion:"1044211", Generation:0, CreationTimestamp:unversioned.Time{Time:time.Time{sec:63603419467, nsec:0, loc:(*time.Location)(0x22f6a20)}}, DeletionTimestamp:(*unversioned.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"heritage":"dockerbuild-test-270779558-19a188ee-617b208a"}, Annotations:map[string]string(nil)}, Spec:api.PodSpec{Volumes:[]api.Volume{api.Volume{Name:"objectstorage-keyfile", VolumeSource:api.VolumeSource{HostPath:(*api.HostPathVolumeSource)(nil), EmptyDir:(*api.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*api.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:ImageID:"docker://daff07e5ebe15908d07159868af56e64ff3d8479670b8551ae834966f9e637e7", ContainerID:"docker://e819b5986f135e0498ceb653ff28de66964b7d117a434276b57f87a58c6db727"}}}})]        
To ssh://[email protected]:2222/test-270779558.git
 ! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'ssh://[email protected]:2222/test-270779558.git'

Interestingly, the k8s log output from the dockerbuilder pod is correct in both cases.

fix(Dockerfile): store the mc alpine binary and download it in Dockerfile

instead of checking it in

dockerbuilder failing to push image on worker node with insecure registry mode disabled

This is because while the builder and controller have the correct credentials for the registry-proxy, dockerbuilder does not. This should be a relatively simple fix to just pass along builder's DEIS_REGISTRY_SERVICE_HOST and DEIS_REGISTRY_SERVICE_PORT environment variables to dockerbuilder.

use git sha for version in the Makefile

this line should change to VERSION ?= git-$(shell git rev-parse --short HEAD)

Use deis-objectstorage-secret.yaml to get S3 credentials

implement dockerbuilder tests

The test target in the Makefile currently does nothing other than display that this project needs tests. This is an official request to add integration tests.

Is this accessing the docker daemon on the build node?

More of a question than an issue. I'm trying to understand if this image is directly accessing the docker daemon on the k8s worker node. How is that possible without elevated privs?

Run docker build on all PR builds

Similar to what builder does in its .travis.yml file, dockerbuilder should run make ... docker-build to ensure that the Dockerfile is correct

Expose DEBUG env variable for for Kubernetes manifests

As a user of dockerbuilder, I'd like to be able to configure the DEBUG level (currently: true vs. false) via a Kubernetes manifest pod ENV variable. See how Builder allows this behavior:

https://github.com/deis/charts/blob/master/deis-dev/manifests/deis-builder-rc.yaml#L35

Do not inject ARG on newer Docker versions.

https://github.com/deis/dockerbuilder/pull/118/files/13b2463d5311da00ac5125534ee7add19a89fa3a#r108909997

I think it might be a good idea to not inject the ARGs in newer Docker versions. For me, each ARG statement takes about 2 seconds to run (Running on Azure). With a bunch of config vars defined, this extends the build time by minutes for me.

As an aside I do wonder why the ARG takes so long when running on my Deis install. Building the Dockerfile directly is super fast. Is this a disk-bound operation?

Builder behaviour differs from Deis 1

I'm using Dockerfile-based deployment in conjuction with Heroku buildpacks. For this purpose I've made my own docker image from the https://github.com/progrium/buildstep. This allows me to run some code with root permissions and then use buildpacks logic for auto-detection. My Dockerfile is as follows (for PHP app):

FROM ineu/buildstep

# Install package with add-apt-repository binary and other needed packages
RUN apt-get update && apt-get install -y software-properties-common unoconv

# Install ffmpeg from the remote repo
RUN add-apt-repository ppa:mc3man/trusty-media && apt-get update && apt-get -y install ffmpeg

ADD . /app
RUN /build/builder

ENV PORT 5000
EXPOSE 5000

CMD ["/start", "web"]

It works great in Deis 1.13. When I try to push the same revision to Deis 2.8, I receive following error:

-----> Preparing runtime environment...
-----> Checking for additional extensions to install...
rm: cannot remove '/app/app/Generator/Abstracts': Directory not emptyrm: cannot remove '/app/app/Generator/Actions': Directory not emptyrm: cannot remove '/app/app/Generator/Interfaces': Directory not emptyrm: cannot remove '/app/app/Generator/Objects/Converters': Directory not emptyrm: cannot remove '/app/app/Generator/Objects/Generators': Directory not emptyrm: cannot remove '/app/app/Resource': Directory not emptyrm: cannot remove '/app/app/config': Directory not emptyrm: cannot remove '/app/tests/Resource': Directory not emptyrm: cannot remove '/app/tests/data': Directory not emptyrm: cannot remove '/app/tmp/images': Directory not empty
The command '/bin/sh -c /build/builder' returned a non-zero code: 1

Are there some known changes in Deis Workflow's dockerbuilder that should be taken into account to migrate Dockerfiles to the new Deis?

Dockerfile apps fail to deploy on beta3

The Docker image isn't successfully pulled from the reigstry and "deis info" shows it as "initializing" forever:

$ kubectl --namespace=yearly-gatepost describe pod yearly-gatepost-v2-cmd-jue7v
Name:       yearly-gatepost-v2-cmd-jue7v
Namespace:  yearly-gatepost
Node:       192.168.64.2/192.168.64.2
Start Time: Tue, 26 Apr 2016 15:05:46 -0600
Labels:     app=yearly-gatepost,heritage=deis,type=cmd,version=v2
Status:     Pending
IP:     172.17.0.15
Controllers:    ReplicationController/yearly-gatepost-v2-cmd
Containers:
  yearly-gatepost-cmd:
    Container ID:   
    Image:      10.100.22.197:80/yearly-gatepost:git-196eb718
    Image ID:       
    Port:       
    QoS Tier:
      cpu:      BestEffort
      memory:       BestEffort
    State:      Waiting
      Reason:       ImagePullBackOff
    Ready:      False
    Restart Count:  0
    Environment Variables:
      DEIS_APP:     yearly-gatepost
      DEIS_RELEASE: v2
Conditions:
  Type      Status
  Ready     False 
Volumes:
  default-token-66ocg:
    Type:   Secret (a volume populated by a Secret)
    SecretName: default-token-66ocg
Events:
  FirstSeen LastSeen    Count   From            SubobjectPath               Type        ReasonMessage
  --------- --------    -----   ----            -------------               --------    -------------
  46s       46s     1   {default-scheduler }                        Normal      Scheduled   Successfully assigned yearly-gatepost-v2-cmd-jue7v to 192.168.64.2
  46s       20s     2   {kubelet 192.168.64.2}  spec.containers{yearly-gatepost-cmd}    Normal      BackOfBack-off pulling image "10.100.22.197:80/yearly-gatepost:git-196eb718"
  46s       20s     2   {kubelet 192.168.64.2}                      Warning     FailedSync  Error syncing pod, skipping: failed to "StartContainer" for "yearly-gatepost-cmd" with ImagePullBackOff: "Back-off pulling image \"10.100.22.197:80/yearly-gatepost:git-196eb718\""

  46s   8s  3   {kubelet 192.168.64.2}  spec.containers{yearly-gatepost-cmd}    Normal  Pulling     pulling image "10.100.22.197:80/yearly-gatepost:git-196eb718"
  46s   8s  3   {kubelet 192.168.64.2}  spec.containers{yearly-gatepost-cmd}    Warning Failed      Failed to pull image "10.100.22.197:80/yearly-gatepost:git-196eb718": Error: image yearly-gatepost:git-196eb718 not found
  46s   8s  3   {kubelet 192.168.64.2}                      Warning FailedSync  Error syncing pod, skipping: failed to "StartContainer" for "yearly-gatepost-cmd" with ErrImagePull: "Error: image yearly-gatepost:git-196eb718 not found"

change the 'name' field in the manifest

this line should have deis-dockerbuilder as its value

implement everything in Go

because of beta crunch time we haven;t explored enough options to use a go cli based client to build docker images and PUSH to remote registry.
Also moving a head in the future we have to support multiple object storage backends.
Also add support to talk to remote docker deamon along with mounting docker socker

Error trying to decode chunk

This stack trace occurred on Workflow 2.1, as reported on community Slack:

(required)>'/usr/bin/bundle:23:in `load'/usr/bin/bundle:23:in `<main>'(See full trace by running task with --trace)        
[stage] remote: Traceback (most recent call last):        
[stage] remote:   File "/deploy.py", line 64, in <module>        
[stage] remote:     log_output(stream, True)        
[stage] remote:   File "/deploy.py", line 16, in log_output        
[stage] remote:     print(chunk.decode('utf-8'))        
[stage] remote: AttributeError: 'dict' object has no attribute 'decode'        
[stage] remote: 2016/07/05 19:14:06 Error running git receive hook [Build pod exited with code 1, stopping build.]        
[stage] To ssh://git@deis-builder

Looking at the code in question, it appears to interpret chunk as a string when it's obviously a dict.

Unit Tests

This project needs some unit tests if we intent on keeping it around as per #81

support SOURCE_VERSION

ala https://devcenter.heroku.com/changelog-items/630

the makefile should use the docker-based workflow

see the prototype repo Makefile for how to do this.

`git push` of `example-dockerfile-http` fails with python backtrace

Push of deis/example-dockerfile-http application to current (2/17/2016) deis-dev manifests.

Looks like deploy.py exploded with backtrace:

k:beef ~ $ kd logs dockerbuild-grassy-sailfish-93bee0e5-84caaf45
download tar file complete
extracting tar file complete
Step 1 : FROM alpine:3.1
Traceback (most recent call last):
  File "/deploy.py", line 50, in <module>
    log_output(stream, True)
  File "/deploy.py", line 15, in log_output
    print(chunk.get('stream').replace('\n',''))
AttributeError: 'NoneType' object has no attribute 'replace'

Full logs: https://gist.github.com/slack/663eb20c5f508be13f3d

pre/post compile hooks

As a Developer using the Deis platfrom I'd like to have pre/post compile hooks available for dockerfile based builds too, so I can execute checks that my services required for tests are alive and execute tests after an image has been made. Thus making my flow more CI/CD enabled.

Disable buffering for docker output

Please disable buffering of the output during a docker build. It can appear as though deploys are hung because output is buffered, for things like bundle install this can be quite frustrating as there isn't much output so the buffer takes a while to fill up.

Transition to ubuntu-slim based image

This repo was missed when most others when through this same transition for rc1. This still needs to get done.