ekino / docker-buildbox Goto Github PK

Objectif: being able to run Percy tests

Feature required:

• Accept env variables to configure the local agent (TOKEN, etc ...)
• Custom script to parse a sitemap.xml file and iterate over the file to analyse pages; this can help: https://github.com/ekino/rendr/blob/master/examples/gatsby/gatsby-node.js#L33-L65
• Able to start a command to init the test suite

Env Variable:

• SITEMAP_XML="https://domain.com/sitemap.xml" - if set, start the sitemap check, for now just let support basic sitemap (not nested definition)
• PERCY_TOKEN=XXXX - mandatory value
• START_COMMAND=percy.js - if set, start the command to run the Percy tests

Hi,

Node 6 has reached its scheduled end-of-life on April 30, 2019. I think it's safe that we stop building Node 6 images.

Hi there!

In order to run some e2e and perf tests I have the need of an image which includes at least one internet browser. Chrome/Chromium seems to be the obvious one as Lighthouse (the perf checker tool I want to use) is based on it.

What do you think about that? Would you consider a PR on the subject?

Hello there,

I would like to suggest some refactoring of docker-buildbox project.

Template engine

Switching from .tpl to jinja2 template, natively supported by Python and avoiding calling sed inside a python script.

One config file to rule them all

External configuration file for variables like extra build arguments or template variables for each image instead of a list of env vars in the CI configuration file.

The goals are both keeping the CI configuration clear and simple AND easing contributors to create/maintain new images.

Each image configuration would look like that:

image_name:
  version:
    cmd_test: [...]
    build_args: [...]
    template_vars: [...]  # if templated Dockerfile
    dockerfile_dir: /path/to/dockerfile  # if Dockerfile's path is not ./<image_name/Dockerfile (ex: java6 image)

Better CLI

travis.py script refactoring using Click, Jinja2 template engine and Python Docker client for a better understandable CLI and no more script editing to add new image.

I have though of something like the following: python travis.py build --image <image_name> --version <image_version>. Everything else would be managed in the configuration file mentioned earlier.

Extra point

I think it would be clearer to manage each image in a different docker repository as it will be easier to create meaningful tags and find the latest for the image we want.

For exemple, creating a repository ekino/<image_name> for each image will make multi-version images easier to manage. This way we will only have to manage tags like <version>-latest or <version>-<tag_date>. For me it will make it way easier to use for newcomers or external people.

I have already spent some time on this refactoring to see if everything could be managed. I still have to work on test commands and prettier output logs but everything works.

Because openjdk docker image doesn't contains latest fix (https://mail.openjdk.java.net/pipermail/jdk8u-dev/2019-May/009330.html), we should move to adoptopenjdk images . ...

Hi! I saw that you're using tfenv in this repository. It's a really good tool, but it does not support OpenTofu and Terragrunt. My team designed a successor of tenv that support Terraform, Terragrunt and OpenTofu. It will be a good idea to migrate into it in due to the growing popularity of OpenTofu.
url: https://github.com/tofuutils/tenv

Over the years we added a lot of configurations and binaries to the java images. Some of them look outdated or useless:

• ENV DEBIAN_FRONTEND=noninteractive LANG=C.UTF-8 is in the base image
• CI Helper ?
• Modd
• Maven considering almost all projects use Gradle (or the Maven wrapper)
• curl -sSL https://salsa.debian.org/debian/mime-support/raw/master/mime.types -o /etc/mime.types I've no idea why it's here

What do you think ?

Hi,

Java 6 has reached its end-of-life a long time ago. It doesn't seem useful to continue supporting it.

What about generate changelog on release process (instead of add new line on each development) ?
@mremi , what is the current release process ? Is there any way to use something like https://pypi.org/project/gitchangelog/, to build and commit new Changelog.md during release process (maybe by travis) ?

Hi,

I'm trying to get a kubernetes cluster with a gitlab runner to build my image and package to AWS. The same gitlab.yml that works with a public shared runner fails with my private runner running on google cloud kubernetes.

I have the gitlab runner to run with privileged mode = true:

runners:
     privileged: true

Here's my gitlab.yml file snippet:

packageApp:
  stage: package
  image: ekino/docker-buildbox:latest-dind-aws
  services:
    - ekino/docker-buildbox:latest-dind-aws
  variables:
    DOCKER_HOST:   "tcp://ekino__docker-buildbox:2375"
    DOCKER_DRIVER:  "overlay2"
  script:
    - sh package-container-to-aws-repo.sh

Here's the output of my build:

Running with gitlab-runner 11.4.0 (8af42251)
  on gitlab-runner-gitlab-runner-89486b67f-gsx8r 2ec769bf
Using Kubernetes namespace: default
Using Kubernetes executor with image ekino/docker-buildbox:latest-dind-aws ...
Waiting for pod default/runner-2ec769bf-project-3107138-concurrent-0zx8sm to be running, status is Pending
Waiting for pod default/runner-2ec769bf-project-3107138-concurrent-0zx8sm to be running, status is Pending
Running on runner-2ec769bf-project-3107138-concurrent-0zx8sm via gitlab-runner-gitlab-runner-89486b67f-gsx8r...
Cloning repository...
Cloning into '/xxxx/xxxx'...
Checking out 034fcdb0 as aws...
Skipping Git submodules setup
Downloading artifacts for buildDotNetServer (116521140)...
Downloading artifacts from coordinator... ok        id=116521140 responseStatus=200 OK token=rk7kgc3a
$ sh package-container-to-aws-repo.sh
Logging into registry.gitlab
Warning: failed to get default registry endpoint from daemon (error during connect: Get http://ekino__docker-buildbox:2375/v1.38/info: dial tcp: lookup ekino__docker-buildbox on 10.55.240.10:53: no such host). Using system default: https://index.docker.io/v1/
error during connect: Post http://ekino__docker-buildbox:2375/v1.38/auth: dial tcp: lookup ekino__docker-buildbox on 10.55.240.10:53: no such host
error during connect: Post http://ekino__docker-buildbox:2375/v1.38/build?buildargs=%7B%7D&cachefrom=%5B%5D&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=Dockerfile&labels=%7B%7D&memory=0&memswap=0&networkmode=default&rm=1&session=m8omnicmei57q265sd7l4wi64&shmsize=0&t=xxxxx&target=&ulimits=null&version=1: dial tcp: lookup ekino__docker-buildbox on 10.55.240.10:53: no such host
Tagging Build to link to AWS registry
error during connect: Post http://ekino__docker-buildbox:2375/v1.38/images/xxxxx/tag?repo=954606123491.dkr.ecr.ap-northeast-1.amazonaws.com%2Fxxxxx&tag=latest: dial tcp: lookup ekino__docker-buildbox on 10.55.240.10:53: no such host
logging on to aws registry
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Warning: failed to get default registry endpoint from daemon (error during connect: Get http://ekino__docker-buildbox:2375/v1.38/info: dial tcp: lookup ekino__docker-buildbox on 10.55.240.10:53: no such host). Using system default: https://index.docker.io/v1/
error during connect: Post http://ekino__docker-buildbox:2375/v1.38/auth: dial tcp: lookup ekino__docker-buildbox on 10.55.240.10:53: no such host
pushing docker
error during connect: Post http://ekino__docker-buildbox:2375/v1.38/images/1234567890.dkr.ecr.ap-northeast-1.amazonaws.com/xxxxxx/push?tag=latest: dial tcp: lookup ekino__docker-buildbox on 10.xx.xx.xx:53: no such host
ERROR: Job failed: command terminated with exit code 1

Any idea what I'm doing wrong?

Hi,
Considering that the official Docker registry now limits the number of downloads for free users.

https://github.blog/2020-09-01-introducing-github-container-registry/

Dind's build on master is currently failing because of a seemingly missing dependency while instaling azure-cli through pip.

The first fail happened during a scheduled build : https://github.com/ekino/docker-buildbox/runs/3529025405?check_suite_focus=true

Collecting jsmin~=2.2.2
Downloading jsmin-2.2.2.tar.gz (12 kB)
    ERROR: Command errored out with exit status 1:
     command: /usr/bin/python3 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-ox9i3x5d/jsmin_12f639f2b015485ebafedb82a9159178/setup.py'"'"'; __file__='"'"'/tmp/pip-install-ox9i3x5d/jsmin_12f639f2b015485ebafedb82a9159178/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-d0awmlgc
         cwd: /tmp/pip-install-ox9i3x5d/jsmin_12f639f2b015485ebafedb82a9159178/
    Complete output (1 lines):
    error in jsmin setup command: use_2to3 is invalid.
    ----------------------------------------

WARNING: Discarding https://files.pythonhosted.org/packages/17/73/615d1267a82ed26cd7c124108c3c61169d8e40c36d393883eaee3a561852/jsmin-2.2.2.tar.gz#sha256=b6df99b2cd1c75d9d342e4335b535789b8da9107ec748212706ef7bbe5c2553b (from https://pypi.org/simple/jsmin/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

INFO: pip is looking at multiple versions of <Python from Requires-Python> to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of pipenv to determine which version is compatible with other requirements. This could take a while.
ERROR: Could not find a version that satisfies the requirement jsmin~=2.2.2 (from azure-cli) (from versions: 2.0, 2.0.1, 2.0.2, 2.0.2.post1, 2.0.3, 2.0.5, 2.0.6, 2.0.7, 2.0.8, 2.0.9, 2.0.10, 2.0.11, 2.1.0, 2.1.1, 2.1.2, 2.1.3, 2.1.4, 2.1.5, 2.1.6, 2.2.0, 2.2.1, 2.2.2)

ERROR: No matching distribution found for jsmin~=2.2.2

For information, we install the same version (2.27.2) in azure image, which is fine.

The difference might be the python or pip version.

From discussion started here, switching for Debian-based images instead of Alpine-based would be a great improvement, at least for all image used for python scripts/app.

Plus it will avoid random failing build caused by Musl lack of compatibility and resiliency.

I will start upgrading these images and clean them (I dont think ci-helper is needed on images only used by ops team).

TODO :

• JDK 11.0.14
• JDK 17.0.2

Tasks

• check if modd and ci-helper work on arm.
• found a clever way to pass the arch name into the Dockerfile so arch dependant binary can be used
• alert the client to use a new attribute platform (https://docker-py.readthedocs.io/en/stable/images.html#docker.models.images.ImageCollection.build)
• define the list of platforms required, intel and arm, but we need to find the technical name
• adjust the matric ci to run the platform
• test each image with the new build

The last version of the AWS image that has make is the 2019.12.26, is that on purpose? The changelog has no info about the removal.

cc @Mallear @clemstoquart

Hi,

Currently our java images are based on images from https://hub.docker.com/u/adoptopenjdk which doesn't seem to be the official repo anymore.

We should use base images from https://hub.docker.com/_/adoptopenjdk instead.

https://hub.docker.com/_/eclipse-temurin

What do you think ?

Hello,

Nodejs 12 will reach EOL in 2 months (30 Apr 2022) and there's no good reason to continue using it.

Hi! Had this working before, but recently getting this error in CI output:

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

This is how gitlab-ci.yml looks like for the stage in question:

deploy:staging:
  image: ekino/docker-buildbox:latest-dind-aws
  stage: deploy
  environment:
      name: staging
  script:
    - ./ci/deploy_api.stage.sh
  only:
    - master
  dependencies:
    - build:staging

deploy_api.stage.sh:

cp _build/stage/rel/api/releases/${VERSION}/api.tar.gz api.tar.gz
docker build -f docker/phoenix/build.df -t ${ECR_API_REPOSITORY}:staging .
eval $(aws ecr get-login)
docker push ${ECR_API_REPOSITORY}:staging

Any idea what might need to be updated if there was a change on gitlab side? I'm using public runner available for CE projects.

I think that we can delete the ci-helper from the java image, gitlab-ci improved cache management, so it's not anymore usefull.
(wait for @Mallear refactoring MR)

Hi,

For some currently unknown reasons Chrome 77 doesn't work on Alpine Linux.

All commands fail with a cryptic error message :

'google-chrome --no-sandbox --version' command failed
Error relocating /usr/lib/chromium/chrome: _ZNSt7__cxx1118basic_stringstreamIcSt11char_traitsIcESaIcEEC1Ev: symbol not found
Error relocating /usr/lib/chromium/chrome: _ZNSt7__cxx1119basic_ostringstreamIcSt11char_traitsIcESaIcEEC1Ev: symbol not found
Error relocating /usr/lib/chromium/chrome: _ZNSt19_Sp_make_shared_tag5_S_eqERKSt9type_info: symbol not found

• Remove node 5.5 it is now replaced by node 6.X
• Update node version
• Update tools version
• Update Changelog

Hello,

Alpine provide a fast way to build image and small image size (vs debian).

Do you have any blocker for this migration ?

Thanks.

Would it be possible to switch the container tagging to use semantic versioning instead of the current YYYY-MM-DD setup?
This makes possible for using tools like Renovate to update the container whenever there is a new release.

Hi,
Does everyone agree ?

#16 did not fix the problem apparently.

Hello, there

I'm opening this issue following the merge of my previous PR #172 .

As discussed in issue #168 and with @Crow-EH in #172, I would like to discuss the possibility to create a Docker Hub repository for each docker image we build.

This way it will be easier to create meaningful tags and find the latest for the image we want, we will be able to give real sementic tags to some image which version hasn't change because it doesn't reflect a tool version (I think of aws or dind-aws image).

I suggest these patterns:

• ekino/<image_name>:< version >-latest/nightly
• ekino/<image_name>:< version >-<tag_date>

for images with multiple versions (java, php, nodes ..)

• ekino/<image_name>:latest/nigthly
• ekino/<image_name>:<tag_date>

for images with one version (aws, ansible, chrome ...)

Hello !

Now that we implemented the dynamic matrix for the build job, what do you think about disabling the fail-fast strategy ? (It's true by default)

It's not necessary anymore for pull requests, and it would allow to successfully build and push images from master even when some images are failing.

See here : https://github.com/ekino/docker-buildbox#dind---aws
Update with new name : ci-dind
Add variables :
DOCKER_TLS_CERTDIR: “”
DOCKER_DRIVER: overlay2
DOCKER_HOST: “tcp://ekino__ci-dind:2375”

As discussed briefly in #621 (comment), react-native is not used anymore and won't be maintained, so we can delete it.

It's installed in one step, and removed because of its dependencies right afterwards:

The following packages will be REMOVED:
  build-essential* ca-certificates* dpkg-dev* fakeroot* file* fop*
  gsfonts-x11* libaether-java* libcommons-vfs-java* libdoxia-java*
  libfop-java* libice6* libmaven-scm-java* libmaven2-core-java*
  libplexus-containers1.5-java* libsisu-ioc-java* libsm6* libssl-dev*
  libwagon-java* libwagon2-java* libx11-6* libx11-data* libxext6* libxi6*
  libxmuu1* libxt6* libxtst6* manpages* manpages-dev* maven* openssl* patch*
  x11-common* xauth* xfonts-encodings* xfonts-utils* xz-utils* zlib1g-dev*

(2nd line from the end)

What do you think about that ?

Hi,

This tools aren't very popular these days.

What do you think about that ?

Hi,

This is mostly to keep github actions up to date.